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RTO  is  the  single  focus  in  NATO  for  Defence  Research  and  Technology  activities.  Its  mission  is  to  conduct  and  promote 
co-operative  research  and  information  exchange.  The  objective  is  to  support  the  development  and  effective  use  of 
national  defence  research  and  technology  and  to  meet  the  military  needs  of  the  Alliance,  to  maintain  a  technological 
lead,  and  to  provide  advice  to  NATO  and  national  decision  makers.  The  RTO  performs  its  mission  with  the  support  of  an 
extensive  network  of  national  experts.  It  also  ensures  effective  co-ordination  with  other  NATO  bodies  involved  in  R&T 
activities. 


RTO  reports  both  to  the  Military  Committee  of  NATO  and  to  the  Conference  of  National  Armament  Directors. 
It  comprises  a  Research  and  Technology  Board  (RTB)  as  the  highest  level  of  national  representation  and  the  Research 
and  Technology  Agency  (RTA),  a  dedicated  staff  with  its  headquarters  in  Neuilly,  near  Paris,  France.  In  order  to 
facilitate  contacts  with  the  military  users  and  other  NATO  activities,  a  small  part  of  the  RTA  staff  is  located  in  NATO 
Headquarters  in  Brussels.  The  Brussels  staff  also  co-ordinates  RTO’s  co-operation  with  nations  in  Middle  and  Eastern 
Europe,  to  which  RTO  attaches  particular  importance  especially  as  working  together  in  the  field  of  research  is  one  of  the 
more  promising  areas  of  co-operation. 


The  total  spectrum  of  R&T  activities  is  covered  by  the  following  7 
•  AVT  Applied  Vehicle  Technology  Panel 
Human  Factors  and  Medicine  Panel 
Information  Systems  Technology  Panel 
NMSG  NATO  Modelling  and  Simulation  Group 
SAS  System  Analysis  and  Studies  Panel 

Systems  Concepts  and  Integration  Panel 
Sensors  and  Electronics  Technology  Panel 


HFM 

1ST 


SCI 

SET 


bodies: 


These  bodies  are  made  up  of  national  representatives  as  well  as  generally  recognised  ‘world  class’  scientists.  They  also 
provide  a  communication  link  to  military  users  and  other  NATO  bodies.  RTO’s  scientific  and  technological  work  is 
carried  out  by  Technical  Teams,  created  for  specific  activities  and  with  a  specific  duration.  Such  Technical  Teams  can 
organise  workshops,  symposia,  field  trials,  lecture  series  and  training  courses.  An  important  function  of  these  Technical 
Teams  is  to  ensure  the  continuity  of  the  expert  networks. 

RTO  builds  upon  earlier  co-operation  in  defence  research  and  technology  as  set-up  under  the  Advisory  Group  for 
Aerospace  Research  and  Development  (AGARD)  and  the  Defence  Research  Group  (DRG).  AGARD  and  the  DRG  share 
common  roots  in  that  they  were  both  established  at  the  initiative  of  Dr  Theodore  von  Karman,  a  leading  aerospace 
scientist,  who  early  on  recognised  the  importance  of  scientific  support  for  the  Allied  Armed  Forces.  RTO  is  capitalising 
on  these  common  roots  in  order  to  provide  the  Alliance  and  the  NATO  nations  with  a  strong  scientific  and  technological 
basis  that  will  guarantee  a  solid  base  for  the  future. 


The  content  of  this  publication  has  been  reproduced 
directly  from  material  supplied  by  RTO  or  the  authors. 
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CTEF  2.0  -  Assessment  and  Improvement  of 
Command  Team  Effectiveness:  Verification 
of  Model  and  Instrument 

(RT  O-TR-HFM- 127) 


Executive  Summary 


Leaders  need  to  gain  and  maintain  better  insight  into  the  effectiveness  of  their  teams  or  staffs.  The  mix  of 
military,  political,  and  societal  objectives  and  constraints,  and  the  increasing  need  for  multi-national,  joint 
military  operations  with  ad-hoc  teams,  even  at  the  lower  command  levels,  has  made  effective  teamwork  a 
critical  mission  success  factor.  The  objective  of  this  study  is  to  support  leaders  and  their  teams  or  staffs  in 
achieving  effective  team  performance.  The  focus  here  is  on  commander-staff  and  commander-sub¬ 
commander  teams  at  operational  and  tactical  command  levels. 


The  CTEF  2.0  model,  developed  from  the 
CTEF  2005  model,  represents  the  scientific, 
empirical  and  theoretical  consensus  that 
effective  teamwork  is  the  result  of  conditions, 
processes,  outcomes  and  feedback  factors. 
The  model  comprises  eleven  components,  such 
as  mission  context  or  team  focused  behaviours, 
each  characterised  by  a  set  of  features,  which 
sums  to  32  in  total.  Based  on  the  CTEF  model, 
an  assessment  instrument  for  commander  and 
staff  teams  was  constructed  in  the  form  of  a 
questionnaire.  Using  this  questionnaire,  the  status 
of  the  team’s  effectiveness  can  be  reviewed, 
which  can  then  be  used  for  improvement  during 
team  evaluations. 


PROCESSES 

OUTCOMES 

The  CTEF  2.0  model  was  developed  based  on  the  experience  and  results  from  applications  in  several 
operational  settings,  and  from  a  survey  version  of  the  instrument  (a  web-based  and  paper-based 
questionnaire),  which  was  administered  in  several  NATO  countries  in  the  period  2008  -  2009.  Respondents 
were  asked  to  reflect  on  the  performance  and  effectiveness  of  a  command  team  they  had  been  part  of  most 
recently,  as  leader  or  member.  The  survey  resulted  in  718  valid  responses  from  14  nations.  The  results 
allowed  us  to  reduce  the  number  of  questions  used  in  the  CTEF  2005  version  to  the  32  of  CTEF  2.0. 
Added  was  the  strong  effect  of  Team  focused  behaviours  on  Task  focused  behaviours,  which  was  not 
predicted  a  priori. 

Operational  usability  of  the  model  and  the  instrument  was  high: 

1)  70%  of  the  respondents  either  “agreed”  or  “completely  agreed”  that  the  instrument  addresses  the 
important  aspects  of  command  teams;  and 

2)  Approximately  80%  of  the  respondents  also  indicated  that  they  would  be  willing  to  use  the  CTEF 
instrument  in  educational,  training,  or  operational  settings. 
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CTEF  2.0  is  now  the  basis  for  further  application  and  data  gathering  in  operational,  exercise,  and  training 
context.  It  is  recommended  that  commanders  learn  to  work  with  the  effectiveness  concepts  proposed  by 
the  CTEF  2.0  model  and  apply  it  regularly,  possibly  selecting  those  elements  they  want  target  for 
improvement.  We  recommend  that  a  next  development  of  CTEF  is  to  include  features  of  more  complex 
organisations,  where  multiple,  more  or  less  independent  teams  and  diverse  organisation  elements  work 
together  to  achieve  common  overarching  goals  (i.e.,  a  team  of  teams). 
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CTEF  2.0  -  Diagnostic  et  amelioration  de  l’efficacite 
d’un  team  de  commandement :  Verification 
du  modele  et  de  l’instrument 

(RT  O-TR-HFM- 127) 


Synthese 


II  est  necessaire  que  les  responsables  aient  une  meilleure  vision  de  Tefficacite  de  leurs  equipes. 
Le  melange  d’objectifs  et  contraintes  militaires,  politiques  et  societaux,  ainsi  que  la  necessity  croissante 
d’operations  militaires  multinationales  interservices  dirigees  par  des  equipes  ad  hoc,  meme  a  des  echelons 
de  commandement  subordonnes,  ont  fait  du  travail  en  equipe  efficace  un  facteur  de  succes  critique  d’une 
mission.  L’objectif  de  cette  etude  est  d’assister  les  responsables  et  leurs  equipes  a  atteindre  un  niveau  de 
prestation  eleve.  Ici,  l’axe  est  mis  sur  des  relations  du  type  commandant-etat-major  et  commandant- 
commandants  subordonnes,  aux  niveaux  de  commandement  tactique  et  operatif. 


Le  modele  CTEF  2.0,  developpe  sur  base  du 
CTEF  2005  reflete  le  consensus  scientifique, 
empirique  et  theorique  prouvant  que  le  travail  en 
equipe  efficace  est  la  resultante  des  facteurs 
conditions,  processus,  resultats  et  compte- 
rendus.  Le  modele  comprend  onze  composantes, 
telles  que  le  Contexte  de  la  Mission  ou  les 
Comportements  focalises  sur  Tequipe.  Chacune 
de  ces  composantes  est  definie  par  un  ensemble 
de  caracteristiques;  32  elements  au  total. 
En  se  basant  sur  ce  modele,  un  instrument 
d’ appreciation  a  ete  construit  qui  prend  la 
forme  d’un  questionnaire.  Ainsi,  le  niveau 
d’efficacite  d’une  equipe  peut  etre  mesure  et 
cette  information  peut  alors  etre  utilisee  pour 
T  amelioration  de  Tefficacite. 


CONDITIONS 

PROCESSES 

OUTCOMES 

Le  modele  CTEF2.0  a  ete  developpe  sur  la  base  des  experiences  et  resultats  recueillis  d’ applications  dans 
differents  contextes  operationnels  et  d’une  enquete  realisee  a  l’aide  d’une  version  web  et  d’une  version 
papier-crayon,  appliquee  dans  plusieurs  pays  OTAN  en  2008  -  2009.  II  etait  demande  aux  participants  de 
reflechir  sur  la  prestation  et  Tefficacite  les  plus  recentes  d’une  equipe  de  commandement  a  laquelle  ils 
appartenaient,  en  tant  que  responsable  ou  en  tant  que  membre.  Quelques  718  reponses  valides  provenant  de 
14  pays  ont  ete  recueillies.  Les  resultats  obtenus  nous  ont  permis  de  reduire  le  nombre  de  questions  du  CTEF 
2005  aux  32  se  rapportant  au  CTEF  2.0.  Outre  des  relations  predites  nous  avons  trouve  une  association  forte 
entre  les  Comportements  focalises  sur  la  Tache  et  les  Comportements  focalises  sur  l’equipe. 

L  ’utilite  operationnelle  du  modele  et  de  l’outil  est  consideree  comme  haute  : 

1)  Quelques  70%  des  repondants  etaient  «  d’ accord  »  ou  «  totalement  d’ accord  »  sur  le  fait  que  l’outil 
met  en  evidence  les  aspects  importants  d’un  team  de  commandement ;  et 

2)  A  peu  pres  80%  mentionnaient  qu’ils  utiliseraient  l’outil  en  situation  operationnelle,  d’entrainement 
et/ou  de  formation. 
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Le  CTEF  2.0  constitue  dorenavant  la  base  pour  des  applications  futures  dans  un  contexte  operationnel, 
d’exercice  ou  de  formation.  II  est  recommande  que  les  commandants  apprennent  a  travailler  avec  les 
concepts  proposes  dans  le  modele  CTEF  2.0  et  l’emploient  regulierement,  tout  en  selectionnant  le  cas 
echeant  les  elements  qu’ils  visent  a  ameliorer.  Nous  recommandons  egalement  qu’un  developpement  futur 
du  CTEF  inclue  les  caracteristiques  d’ organisations  plus  complexes,  dans  lesquelles  une  multitude  de  teams 
plus  ou  moins  independants  et  d’autres  elements  organisationnels  travaillent  ensemble  pour  realiser  des 
objectifs  communs  d’ordre  superieur  (c.a.d.  des  teams  composes  de  teams). 
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1.1  BACKGROUND 

Since  the  end  of  the  Cold  War,  military  missions  have  become  increasingly  complex.  Military  commanders 
are  confronted  with  a  broadening  range  of  mission  types,  ranging  from  humanitarian  relief,  to  peace  support 
operations,  to  traditional  warfare.  In  addition,  they  are  also  held  accountable  by  a  wider  range  of 
stakeholders,  potentially  representing  diverse  objectives:  societal,  political,  economic,  and  military. 
Furthermore,  operating  in  multi-national  settings  with  ad  hoc  task  forces  composed  of  multiple  services  and 
multiple  countries  has  become  normal  practice.  Covering  larger  operational  areas  with  fewer  forces  and 
quickly  changing  operational  conditions  require  commanders  to  delegate  to  the  lower  levels  of  command. 
In  these  complex  contexts  commanders  must  rely  on  the  effective  interaction,  communication, 
and  collaboration  of  their  staffs,  sub-commanders,  and  other  civilian  and  military  parties  outside  the  chain  of 
command. 

The  reality  of  ad  hoc,  multi-national,  joint  military  and,  increasingly,  multi-agency  teams  in  current 
operations  has  brought  forward  issues  such  as  team  maturity,  leadership,  cultural  diversity,  and  organisational 
support  in  the  context  of  operational  stress,  high  stakes  missions,  diverse  command  structures,  and  their 
impact  on  the  effectiveness  of  the  team.  Increasingly  mission  success  will  rely  on  how  well  commanders 
understand  the  strengths  and  weaknesses  of  their  teams,  and  how  well  they  can  improve  their  effectiveness. 
Knowledge  of  the  factors  that  affect  the  success  of  command  and  staff  teams  will  help  to  build  effective 
command  teams,  select  the  best  personnel,  improve  organisational  structures,  rapidly  develop  team 
arrangements,  and  optimise  processes  in  operational  conditions,  all  in  the  context  of  mission  goals  and 
stakeholders  expectations. 

Building  and  maintaining  effective  teams  is  a  continuous  process.  Commanders  and  team  members  need  to 
assess,  control,  and  adjust  the  qualities  and  maturity  of  the  team  before  and  during  the  mission,  in  order  to 
achieve  intermediate  and  end  goals.  In  addition,  after  a  mission  team  processes  and  outcomes  need  to  be 
reviewed  in  order  for  the  organisation  to  learn  from  the  experience  for  future  missions.  To  support  these 
efforts  a  NATO  RTO  HFM  Task  Group  (HFM-087)  developed  a  comprehensive  model  for  Command  Team 
Effectiveness  (CTEF)  and  a  corresponding  instrument  to  meet  these  concerns  (Essens  et  al.,  2005). 
The  CTEF  model  was  based  on  an  extensive  literature  review  on  team  effectiveness  and  analysis  of 
operational  command  teams.  This  model  is  discussed  in  more  detail  in  the  report  of  HFM-087  (Essens  et  al., 
2005)  and  is  briefly  summarized  below.  A  subsequent  NATO  RTO  HFM  Task  Group  (HFM-127)  - 
the  activities  of  which  are  reported  here  -  applied  the  CTEF  model  and  instrument  in  operational  conditions 
with  the  purpose  of  empirically  verifying  the  operational  value  and  usefulness  of  the  instrument  for 
commanders  and  their  teams. 

1.2  GOAL  OF  THE  STUDY 

The  primary  goal  of  the  study  was  to  examine  the  operational  utility  and  verify  the  theoretical 
underpinnings  of  the  model-based  instrument  in  the  light  of  empirical  data.  We  investigated  the  conceptual 
consistency  with  the  existing  team  effectiveness  research  literature  and  psychometric  properties  of  the 
instrument.  Adjustments  to  the  instrument  were  based  on  these  empirical  findings.  The  intention  was  to 
arrive  at  a  well-grounded  instrument  that  supports  commanders  and  their  teams  in  achieving  effective 
team  and  task  processes  for  operational  effectiveness. 

1.3  TARGET  AUDIENCE 

The  CTEF  model  and  instrument  is  primarily  intended  for  military  commanders  and  team  members  who 
want  to  improve  their  team  performance  and  effectiveness.  The  CTEF  instrument  is  intended  to  be 
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generically  applicable  to  command  teams  at  staff  levels  independent  of  configuration,  including  multi¬ 
service,  multi-agency,  and  international  teams,  co-located  or  distributed.  Command  teams  are 
characterised  by  a  high  degree  of  dynamic  interaction  between  team  members  directed  to  situation 
understanding,  problem  solving,  planning,  and  decision  making.  In  current  operations,  these  teams  may  be 
standing  teams  or  may  be  formed  on  an  ad-hoc  basis,  focused  on  specific  tasks  or  missions  requiring 
concentrated  efforts  with  diverse  competencies  and  capabilities.  These  missions  may  last  for  short  or  long 
periods  of  time  (months  to  years)  with  potential  turnover  of  personnel.  The  CTEF  instrument  can  be  used 
by  commanders,  by  command  teams  or  staffs,  by  trainers  and  evaluators  to  perform  assessments  of  the 
effectiveness  of  operational  teams  or  staffs,  or  by  scientists  to  collect  data  on  team  effectiveness  in 
operational  conditions. 


1.4  METHOD  OF  WORK 

The  evaluation  and  verification  process  of  the  model  and  instrument  was  performed  in  two  phases: 
application  in  operational  exercise  settings  and  administration  of  an  international  survey.  First,  we  applied 
the  CTEF  instrument  in  several  operational  exercise  settings  to  gain  insights  into  the  instrument  itself, 
with  particular  emphasis  on  the  ease  and  feasibility  of  administering  the  instrument  (e.g.,  intelligibility  of 
the  items,  interpretation  of  the  scales,  time  to  complete).  In  order  to  demonstrate  the  value  of  CTEF  for 
commanders  to  continuously  improve  their  team  we  closed  the  assessment  loop.  In  addition  to  just 
collecting  data  with  the  instrument,  we  developed  a  rapid  cycle  of  collecting  and  processing  data  with  the 
CTEF  instrument,  and  provided  summary  results  to  commanders  that  could  be  used  as  feedback  to  their 
teams  while  in  operations.  Administrations  of  the  instrument  resulted  in  some  changes  in  terms  of  reduced 
detail  (fewer  items  in  several  sections),  and  less  complex  wording  of  the  items.  We  also  developed  a 
template  for  feedback  to  commanders.  To  increase  the  ease  of  administration  and  speed  the  reporting  of 
results  (within  two  days)  we  developed  a  computer-based  version  of  the  instrument  and  standardized  the 
statistical  analysis  routines. 

In  the  second  phase,  we  administered  the  instrument  in  a  survey  format  in  14  NATO  countries.  The  survey 
was  directed  to  commanders  and  team  members  who  have  been  part  of  a  command  team  in  a  recent 
deployment  or  other  operational  context.  The  resulting  data  (718  respondents)  were  used  to  verify  the 
theoretical  underpinnings  of  the  CTEF  model  and  to  identify  additional  ways  to  enhance  the  utility  of  the 
CTEF  instrument  (e.g.,  by  further  reducing  the  number  of  items  or  responses  required). 

We  operated  as  Task  Group  HFM-127  in  the  context  of  the  Human  Factors  and  Medicine  Panel  of  the 
NATO  Research  and  Technology  Organisation.  The  Group  consisted  of  representatives  from  Canada, 
USA,  Belgium,  Sweden,  and  the  Netherlands,  all  with  a  strong  background  in  military-related  research  in 
command  and  control,  team  performance,  leadership,  and  decision-making  (the  Task  Group  also  benefited 
from  the  contributions  of  researchers  in  Bulgaria  and  the  UK).  The  Task  Group  worked  from  Spring  2005 
to  Spring  2009,  during  which  we  met  eight  times  and  processed  operational  and  survey  data,  as  well  as 
wrote  the  report. 


1.5  STRUCTURE  OF  THE  REPORT 

In  Chapter  2  we  provide  a  concise  overview  of  the  CTEF  model  and  the  version  of  the  instrument 
developed  by  the  earlier  NATO  Task  Group  (‘CTEF  2005’;  Essens  et  al.,  2005).  This  is  intended  to 
provide  the  reader  sufficient  context  for  understanding  the  theoretical  background  of  the  instrument. 
In  Chapter  3  we  discuss  several  applications  of  the  instrument  in  operational  practice  and  the  subsequent 
optimisation  of  the  structure  of  the  instrument.  In  Chapter  4  we  examine  the  conceptual  consistency  of  the 
CTEF  model  with  a  model  fit  test,  and  we  explored  the  psychometric  properties  of  the  instrument  based  on 
the  survey.  The  survey  administration  and  results  are  analysed  and  discussed.  In  Chapter  5  the  survey  data 
are  compared  with  data  of  an  administration  of  the  questionnaire  during  a  mission.  Chapter  6  provides  an 
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overview  of  the  results  and  draws  conclusions  leading  to  revisions  of  the  model  and  the  instrument: 
‘CTEF  2.0’.  In  Chapter  6  recommendations  are  presented.  The  Annexes  present:  Annex  A  -  The  CTEF 
Instrument  based  on  CTEF  2005  model  used  in  Joint  Caribbean  Lion;  Annex  B  -  The  CTEF  Instrument  as 
employed  in  the  multi-national  survey  Study;  Annex  C  -  Demographic  frequencies  for  participants  in  the 
CTEF  survey  study;  Annex  D  -  The  CTEF  2.0  Concepts  resulting  from  the  survey;  Annex  E  -  The  CTEF 
2.0  Instrument;  and  Annex  F  -  Practical  Guidelines  for  application  of  CTEF  instrument. 
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Chapter  2  -  MILITARY  COMMAND  TEAM 
EFFECTIVENESS:  OVERVIEW 

2.1  THE  COMMAND  TEAM  EFFECTIVENESS  MODEL  (CTEF) 

The  NATO  RTO  HFM  Task  Group  (HFM-087)  developed  a  comprehensive  model  for  Command  Team 
Effectiveness  (CTEF)  and  a  model-based  instrument  to  address  the  identified  need  of  commanders  to 
assess  and  improve  the  functioning  of  their  teams  before  and  during  the  mission  (‘CTEF  2005’;  Essens 
et  al.,  2005).  The  CTEF  model  was  based  on  an  extensive  literature  review  of  team  effectiveness  and 
analyses  of  operational  command  teams. 

The  premise  of  the  model  is  that  the  team  to  be  effective,  commanders  must  have  a  clear  understanding  of: 

a)  The  conditions  they  are  working  in  (i.e.,  mission  context,  assigned  task(s),  degree  of  organisational 
support,  and  capabilities  of  team  members,  team  leader,  the  team); 

b)  The  task  and  team  processes  that  are  needed  to  perform  the  mission  and  the  tasks,  given  those 
conditions;  and 

c)  The  objective  end  goals  and  intermediate  goals,  and  the  criteria  for  assessing  the  progress  towards 
these  goals  (i.e.,  task  and  team  output). 

Effective  commanders  regularly  review  the  task  and  team  processes  against  intermediate  outcomes, 
and  adjust  these,  or  even  seek  to  adjust  condition  factors,  if  possible.  The  CTEF  model  helps  the  commander 
to  address  the  relevant  factors  in  the  control  and  improvement  of  effectiveness.  The  model  will  stimulate 
awareness  of  critical  variables  that  may  affect  the  team’s  effectiveness. 

Figure  2-1  shows  the  original  CTEF  model  with  the  top-level  components  of  team  effectiveness,  their 
relationships,  and  feedback  loops. 


CONDITIONS  PROCESSES  OUTCOMES 


Figure  2-1:  ‘CTEF  2005’  Model  with  its  Constituting  Components 
and  Feedback  Loops  (Essens  et  al.,  2005). 
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The  Conditions  component  comprises  one  set  that  addresses  the  context  the  team  is  working  in:  Mission 
Framework,  which  captures  the  conditions  of  the  mission  of  the  team  (e.g.,  situational  uncertainty,  stress 
potential,  limiting  constraints,  and  stakes);  Task,  which  captures  the  goals  the  team  has  to  accomplish, 
and  the  characteristics  of  the  work  that  will  be  involved  (e.g.,  task  complexity,  workload,  goal  clarity, 
and  goal  stability);  and  Organisation,  which  captures  issues  related  to  the  fact  that  the  team  is  embedded 
within  the  context  of  a  broader  organisation  (e.g.,  goal  congruity,  clarity  of  command  structure,  autonomy, 
and  organisational  support). 

A  second  set  of  Conditions  addresses  the  people  that  form  the  team:  Leader  and  Team  Members,  which 
capture  critical  characteristics  that  potentially  affect  the  functioning  and  effectiveness  of  the  team  (skills 
and  knowledge,  and  congruity  of  personal  goals  and  organisational  goals);  and  Team,  which  captures 
critical  characteristics  of  the  team  as  a  unit  (composition,  size,  architecture,  maturity,  and  team  goals). 

The  Processes  are  focused  on  the  behaviours  that  emerge  during  the  operation.  We  use  the  term  processes, 
because  these  are  on-going  activities.  The  relevant  components  address:  Task-Focused  Behaviours,  which 
cover  the  activities  directed  at  the  operational  tasks  (e.g.,  managing  information,  assessing  the  situation, 
making  decisions,  planning,  directing  and  controlling,  and  liaising  with  other  teams);  Team-Focused 
Behaviours,  which  address  the  interactions  between  the  team  members  (e.g.,  providing  and  maintaining 
vision,  maintaining  common  intent,  interacting  within  the  team,  motivating,  adapting  to  changes,  and 
providing  team  maintenance). 

The  Outcomes  address  the  standards,  goals,  criteria,  intentions,  expectations,  and  products  that  are 
explicitly  or  implicitly  intended  results  of  the  mission.  Mission  success  is  not  only  specified  in  military 
terms.  Current  operations  usually  have  multiple  stakeholders,  which  should  be  considered  in  evaluating 
the  effectiveness  of  the  team.  For  the  team,  Outcomes  focus  on  those  measures  that  reflect  mature  and 
potentially  well-performing  teams.  Missions  are  usually  not  a  simple  ‘one-shot’  effort,  but  develop  with 
successive  rotations  and  last  for  longer  periods.  Therefore,  it  is  important  for  commanders  and  teams  to 
specify  and  assess  intermediate  results.  These  can  be  used  for  more  or  less  formal  iterations  in  the  mission 
or  development  of  the  team. 

The  relevant  Outcome  components  and  aspects  are:  Task  Outcomes,  which  address  the  intermediate  or 
final  results  of  the  mission  (e.g.,  achievement  of  the  (intermediate)  goals  or  achievement  of  the  mission, 
meeting  criteria  set  by  stakeholders,  other  stakeholders’  satisfaction);  and  Team  Outcomes,  which  address 
the  maturity  of  the  team  (e.g.,  mutual  trust,  morale,  cohesion,  collective  confidence  in  achieving  the  goals, 
shared  vision,  and  mutual  respect). 

The  model  shows  arrows  between  the  components,  which  indicate  that  the  components  affect  each  other: 
conditions  affect  processes  and  processes  affect  outcomes.  The  Feedback  loops  represent  iterative 
development,  adjustment,  and  learning  processes,  which  follow  (more  or  less)  formal  reviews  of  the 
progress  of  the  team  against  the  outcomes.  [Note:  intrinsic  feedback  processes  may  be  present  within  the 
team-  and  task-focused  behaviours  as  a  natural  element  in  performing  tasks  and  providing  feedback  when 
working  together.  The  Feedback  loops  specify  more  explicitly  the  reviewing  activity]. 

The  specified  Feedback  loops  are:  Process  Adjustment  Loop,  which  addresses  the  required  interventions  in 
the  management  or  performance  in  the  task  and  team  processes;  Conditions  Adjustment  Loop,  which 
addresses  the  changes  needed  in  the  structural  basis  of  the  Conditions,  either  in  personnel,  organisation, 
or  mission  and  task  factors;  and  Organisational  Learning  Loop,  which  addresses  the  evaluation  of  all 
components  of  the  effectiveness  in  the  light  of  the  success  and  failures  in  the  mission;  for  the  commander 
and  team’s  own  learning  cycle,  and  the  advice  to  the  organisation  and  follow  up  commanders.  Table  2-1 
provides  an  overview  of  the  main  components,  aspects  and  features  of  the  CTEF  2005  model,  the  first 
version  of  CTEF.  The  developmental  details  that  led  to  changes  in  wording  which  was  used  in  the  Survey 
version  of  CTEF  are  described  in  Chapter  3. 
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Table  2-1:  Main  Categories,  Components,  Aspects,  and  Features 
of  the  CTEF  Model,  Respectively  the  Levels  1-4. 


CONDITIONS 
(Level  1) 

Mission  Framework  (Level  2) 

Task 

Situational  uncertainty  (Level  3) 

Task  complexity 

Uncertainty  about  intelligence  (Level  4) 

Task  difficulty 

Uncertainty  about  adversary’s  intent 

Number  of  sub-tasks 

Uncertainty  about  required  resources 

Sub-task  interdependencies 

Uncertainty  about  utilisation  of  resources 

Sub-task  interference 

Stress  potential 

Workload 

Danger 

Physical  workload 

Operational  intensity 

Cognitive  workload 

Psychological  stressors 

Emotional  workload 

Constraints 

Time  pressure 

Environmental  factors 

Lack  of  goal  clarity 

Political  factors 

Cultural  factors 

Media  related  factors 

Time-space  coordination 

Stakes 

Casualties 

National  impact 

International  impact 

Lack  of  goal  stability 

Organisation 

Leader 

Congruity  of  team’s  mission  and  organisational  goal 

Leader  skills 

Clarity  of  command  structure 

Tactical  skills 

Autonomy 

Technical  skills 

Organisational  support 

Interpersonal  skills 

Recognition 

Cognitive  skills 

Supportive  climate 

Leader  knowledge 

Material  support 

Task  knowledge 

Team  knowledge 

Organisational  knowledge 

Match  of  personal  goals  to  organisational  goals 

Team  Member 

Team 

Team  member  skills 

Appropriateness  of  team  composition 

Tactical  skills 

Mix  of  skills 

Technical  skills 

Mix  of  demographic  characteristics 

Interpersonal  skills 

Mix  of  personality  traits 

Cognitive  skills 

Membership  stability  over  time 

Team  member  knowledge 

Appropriateness  of  team  size 

Task  knowledge 

Appropriateness  of  team  architecture 

Team  knowledge 

Physical  proximity 

Organisational  knowledge 

Appropriate  distribution  of  tasks 

Match  of  personal  goals  to  organisational  goals 

Team  maturity 

Match  of  team  goals  to  organisational  goals 
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PROCESSES 

Task-Focused  Behaviours 

Team-Focused  Behaviours 

Managing  information 

Providing  and  maintaining  vision 

Obtaining  information 

Maintaining  common  intent 

Processing  information 

Interacting  within  the  team 

Exchanging  information 

Communicating 

Assessing  the  situation 

Coordinating 

Making  decisions 

Providing  feedback 

Defining  the  problem  space 

Motivating 

Managing  time  available  to  make  decisions 

Extrinsically 

Evaluating  options  and  results 

Intrinsically 

Planning 

Adapting 

Anticipating 

Monitoring 

Scheduling 

Correcting 

Specifying  resources  needed 

Backing-up 

Defining  strategies 

Providing  team  maintenance 

Directing  and  Controlling 

Providing  social  support/integration 

Organising 

Regulating  emotions 

Managing 

Developing/maintaining  cohesion 

Monitoring  progress 

Managing  conflict 

Liaising  with  other  command  teams 

OUTCOMES*) 

Task  Outcomes 

Team  Outcomes 

Achievement  of  the  intermediate/end  goals 

Mutual  trust 

Meeting  the  Criteria  set  by  the  stakeholder 

Morale 

Other  stakeholders’  satisfaction  with  (expected) 

Cohesion 

outcomes 

Collective  confidence  in  achieving  the  goal 

Staying  within  the  limits/intentions 

Shared  vision 

Mutual  respect 

FEEDBACK**) 

Processes  Adjustment 

Conditions  Adjustment 

*)  For  outcomes  a  distinction  was  made  between  ‘Achievement  of  Intermediate  goals’  for  assessments  during 
an  operation;  and  of  ‘Achievement  of  End  Goals’  for  assessments  at  the  end  or  after  an  operation. 


**)  For  Feedback  only  the  Level  2  items  (Processes  and  Conditions  Adjustment)  were  specified  in  the  CTEF 
2005  model. 


The  CTEF  2005  model  provided  the  basis  for  an  assessment  instrument  with  items  that  are  organised 
along  the  structure  of  the  model.  The  instrument  allows  the  respondent  to  indicate  the  perceived  level  of 
an  item  and  the  impact  that  level  has  on  team  effectiveness.  We  distinguished  between  actual  status  and 
potential  impact  of  that  status  on  the  eventual  effectiveness  and  ask  the  commander  to  assess  the  factors  in 
two  steps:  First,  an  objective  judgement  is  required  concerning  the  status  of  a  particular  aspect  of  a 
component.  Then,  a  judgement  is  required  to  assess  the  potential  impact  of  that  status  on  team 
effectiveness.  In  the  Figure  2-2  an  example  is  presented  (from  Essens  et  al.,  2005)  with  the  main  category 
Processes  and  component  Task-focused  behaviours  and  one  of  the  aspects  and  the  accompanying  features. 
The  items  show  an  aspect  level  “Managing  Information”  (Fevel  3)  and,  four  features  (Fevel  4).  Each  item 
is  scored  on  the  magnitude1  scale  Very  Fow  to  Very  High  (1  -  5),  or  Not  Available  (i.e.,  N.A.). 
The  impact  of  that  magnitude  rating  on  overall  team  effectiveness  is  scored  on  Very  Negative  to  Very 
Positive  (-2  to  +2). 


1  Throughout  the  text  of  this  report  we  use  the  terms  magnitude,  level,  quantity  interchangeably  to  indicate  the  assessment  of 
status  of  a  given  construct. 
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TASK-FOCUSED  BEHAVIOURS 

Task-related  processes  include  both  productive  and  corrective  behaviours:  managing  information,  assessing  the  situation,  making  decisions, 
planning,  directing  and  controlling,  monitoring  progress,  and  liaising  with  other  command  teams. 


Assessment  of  Task-Focused  Behaviours 

Impact  on  Team  Effectiveness 

Item 

N.A. 

Very 

Low 

Low 

Moderate 

High 

Very 

High 

Very 

Negative 

Negative 

None 

Positive 

Very 

Positive 

Managing  Information 

1 

2 

3 

4 

5 

-2 

-i 

0 

+i 

+2 

Obtaining  Information 

(i.e.,  through  an  active  search  to 
compensate  for  the  lack  of 
information  addressed  under 
situational  uncertainty) 

1 

2 

3 

4 

5 

-2 

-i 

0 

+i 

+2 

Processing  Information 

(e.g.,  using,  integrating 
information) 

1 

2 

3 

4 

5 

-2 

-i 

0 

+i 

+2 

Exchanging  information 

(e.g.,  timeliness,  clarity,  brevity, 
correctness,  completeness) 

1 

2 

3 

4 

5 

-2 

-i 

0 

+i 

+2 

Assessing  the  Situation 

(e.g.,  perceiving,  recognising, 
anticipating  events) 

1 

2 

3 

4 

5 

-2 

-i 

0 

+i 

+2 

Figure  2-2:  Sample  of  CTEF  Instrument  on  the  Basis  of  the  CTEF  (2005)  Model. 


The  CTEF  instrument  can  be  applied  at  various  stages  of  a  team’s  mission  to  assess  the  status  of  the  team 
at  a  given  time.  The  instrument  can  be  used  to  diagnose  the  team’s  conditions,  processes,  and  intermediate 
outcomes  in  the  preparation  of  and  during  a  mission  and  identify  which  elements  require  improvement  in 
order  to  improve  effectiveness.  The  results  and  the  required  improvements  can  be  discussed  by  the 
commander  and  the  team  in  their  regular  team  Progress  Review  sessions.  The  results  can  also  as  a 
benchmark  against  which  the  impact  of  an  intervention  can  be  evaluated.  After  the  team’s  mission, 
the  results  can  be  used  as  a  basis  for  an  After- Action  Review  (AAR),  to  learn  and  to  understand  how  and 
why  things  happened  as  they  did,  and  how  things  can  be  improved.  The  instrument  can  be  applied 
repeatedly  to  reveal  how  the  team  is  evolving  and  performing  over  time.  The  development  and  initial 
applications  of  the  CTEF  instrument  are  described  in  Chapter  3. 
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Chapter  3  -  DEVELOPMENT  AND  INITIAL 
APPLICATIONS  OF  THE  INSTRUMENT 

3.1  INTRODUCTION 

In  this  chapter,  we  describe  initial  tests  and  applications  and  the  subsequent  fine-tuning  of  the  CTEF 
instrument.  The  assessment  instrument  from  the  Essens  et  al.  (2005)  study  was  constructed  as  a 
straightforward  representation  of  the  model,  without  tests  in  operational  conditions.  Two  lines  of  testing 
and  verification  of  the  CTEF  2005  model  and  corresponding  instrument  were  initiated:  we  looked  for 
operational  applications  in  training  and  exercises;  and  an  international  survey  study  was  planned.  Several 
opportunities  arose  where  we  could  apply  the  instrument,  explore  data,  and  evaluate  how  commanders  use 
the  instrument  to  assess  their  teams.  On  the  basis  of  these  data  we  improved  the  instrument  while 
maintaining  the  CTEF  structure.  The  international  survey,  described  in  Chapter  4,  was  intended  to 
evaluate  the  operational  relevance  of  the  model  on  the  basis  of  formulated  hypotheses. 

In  Chapter  3  we  describe  the  initial,  ad  hoc  experiences  with  the  model  and  the  instrument,  which  resulted  in 
several  key  changes  (e.g.,  format,  intelligibility,  content,  scales,  data  analysis,  generation  of  feedback). 
Second,  we  describe  a  substantial  application  of  the  CTEF  model  and  instrument  in  an  operational  exercise 
(Section  3.5  Joint  Caribbean  Lion).  In  our  search  for  opportunities  to  collect  data,  we  were  especially 
interested  in  real  operations  and  exercises  in  the  field.  However,  other  data  sources  were  sampled, 
if  available,  in  order  to  get  feedback  on  the  model  and  the  instrument.  The  only  common  requirement  in  all 
cases  described  below  was  that  respondents  needed  to  have  an  operational  background  and,  preferably, 
some  experience  with  deployments.  The  initial  stage  of  verification  of  the  model  and  the  instrument  was 
subject  to  a  number  of  limitations.  First,  the  number  of  participants  per  case  varied  from  twenty  to  fifty 
persons  only.  Second,  there  was  a  lot  of  variation  in  settings,  in  type  and  composition  of  the  teams,  and  even 
in  measurement.  After  each  use  of  the  instrument,  we  made  revisions  according  to  the  feedback  that  we 
received. 

The  chapter  is  organised  around  four  cases:  the  fourth  Multi-National  Exercise  (MNE-4)  with  teams  at  the 
Canadian  site;  multi-national  teams  at  the  headquarters  of  Supreme  Allied  Command  Transformation 
(SACT);  the  naval-air  exercise  Gammel  Dansk  of  the  Dutch  Naval  Operational  School,  and  the  sea-based 
amphibious  exercise  Joint  Caribbean  Lion  (JCL).  These  cases  allowed  us  to  get  feedback  from  the  military 
on  their  appreciation  of  the  model  and  use  of  the  instrument  as  a  whole,  and  to  uncover  the  qualities  and 
weaknesses  of  the  instrument.  In  these  cases  we  studied  how  teams  scored  the  items.  The  naval  exercise 
Gammel  Dansk  provided  us  with  evaluation  data  on  the  instrument.  The  exercise  Joint  Caribbean  Lion 
provided  longitudinal  data  from  team  leaders,  team  members,  and  observers  from  which  we  could  learn 
how  teams  developed  during  an  exercise. 

Each  case  in  this  chapter  is  described  in  four  sections: 

1)  The  context  of  the  case  (e.g.,  real  operation  or  exercise,  type  of  operation,  number  of  nations 
involved); 

2)  The  method  of  data  collection  (i.e.,  participants,  type  of  instrument  used,  design); 

3)  The  results  of  the  data  analysis  (i.e.,  statistics,  feedback  from  the  field);  and 

4)  The  lessons  learned  to  further  improve  the  instrument. 

The  point  of  departure  for  the  studies  in  this  chapter  was  the  CTEF  2005  model  as  shown  earlier  in  Table 
2-1. 


We  distinguished  between  three  versions  of  the  instrument: 
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•  The  long  version  encompassing  all  four  levels  of  the  model  which  comprise  items  of  the  aspects 
and  the  underlying  features  (see  Table  2-1);  e.g.,  there  are  questions  relating  to  Situational 
Uncertainty  (aspect  level  of  items  -  Level  3)  and  to  Uncertainty  about  required  resources 
(the  feature  level  -  Level  4); 

•  The  medium  version  consists  of  items  of  all  aspects  and  the  underlying  features  (Level  4)  for  the 
processes,  but  only  the  aspect-items  (Level  3)  for  the  conditions  and  the  outcomes;  and 

•  The  short  version  only  contains  three  levels  of  the  model  comprising  the  items  of  the  aspects  for 
the  conditions,  outcomes,  as  well  as  the  processes. 


3.2  MULTI-NATIONAL  EXERCISE-4  (MNE4) 

Our  first  data  collection  attempt  with  the  CTEF  instrument  was  in  the  context  of  the  Multi-National 
Experiment-4  (MNE4;  February-March  2006).  Attempts  were  made  to  coordinate  data  collection  with  CTEF 
at  different  sites  within  the  MNE4  study.  In  the  end,  however,  data  were  collected  only  at  the  Canadian 
location  at  the  Canadian  Forces  Experimentation  Centre  (CFEC,  Ottawa,  Canada),  and  involved  a  limited 
number  of  participants.  Accordingly,  the  results  of  this  study  with  respect  to  the  development  of  the  CTEF 
instrument  were  limited  to  structural  changes  and  usability  feedback  from  subject  matter  experts. 

3.2.1  Context 

The  MNE4  experiment  was  the  4th  study  in  a  series  of  large  technology  demonstration  experiments 
examining  the  robustness  of  the  Effects  Based  Operations  (EBO)  process,  as  well  as  organizational, 
technology,  and  supporting  concepts.  The  countries  involved  were  Australia,  Canada,  Finland,  France, 
Germany,  Sweden,  UK,  and  USA.  Canada  focused  on  Knowledge  Management  and  MNIG  (multi¬ 
national  inter-agency  group),  cultural  analysis,  knowledge-based  development,  and  common  intent. 
The  NATO  Coalition  Task  Force  (CTF)  and  NATO  Response  Force  (NRF)  provided  two  headquarters 
staffs,  both  utilizing  EBO  to  address  a  stability  operations  scenario  in  current  day  Afghanistan. 

The  experiment  was  focused  at  the  operational  level  of  command.  The  traditional  staff  structure  was  not 
employed;  instead  each  HQ  was  functionally  organized  with  a  Command  Group  supported  by  four  multi¬ 
disciplinary  staff  teams  based  on  the  EBO  framework:  Effects  Based  Planning,  Effects  Based  Execution, 
Effects  Based  Assessment,  and  Knowledge  Superiority.  Both  HQs  were  supported  by  a  Multi-National 
Inter-agency  Group,  composed  of  representatives  from  other  organizations  and  NGOs.  The  CTF  HQ 
operated  in  a  distributed  fashion  from  five  nations,  communicating  via  the  Combined  Federated  Battle  lab 
Network.  The  NRF  HQ  personnel  were  located  in  Istanbul,  Turkey,  and  also  communicated  via  the 
Combined  Federated  Battle  lab  Network. 

3.2.2  Method 

The  structure  of  the  instrument  was  modified  to  address  more  clearly  the  positive  or  negative  impact  of  a 
particular  magnitude  of  an  item  on  the  team  effectiveness.  The  separate  positive  or  negative  direction 
replaced  the  -2  (Very  negative)  to  +2  (Very  Positive)  scale  of  the  original  instrument  (Figure  3-1). 
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TO  WHAT  DEGREE  ARE  EACH  OF  THESE 
ELEMENTS  PRESENT  IN  YOUR  COMMAND 
TEAM? 

Rel 

Quality/Magnitude 

Direction 

Strength  of  Impact 

Situational  Uncertainty 

The  lack  of  information  about,  or  lack  of  understanding  of  objects  and 
their  properties  in  the  area  of  responsibility 

® 

®  ©®  ©@ 

o®  © 

®  ©@  ©® 

Stress  Potential 

Stress  potential  due  to  operational  consequences 

©  ©®  ©© 

0@© 

@0©©® 

Constraints 

External  factors  that  limit  the  range  of  the  team ’s  actions  or  autonomy 

®  O®  ©® 

0®  © 

@®@®® 

Figure  3-1 :  Part  of  the  Questionnaire  that  was  used  in  MNE4. 


The  Canadian  MNE4  location  at  CFEC  in  Ottawa  had  a  total  of  14  ‘players’;  1 1  of  them  participated  in  the 
survey.  The  CTEF  instrument  was  administered  three  times  during  the  course  of  the  3-week  study  -  on  days 
4,  10,  and  15.  The  response  rate  for  the  three  administrations  was  9,  11,  and  11  participants,  respectively. 
Due  to  time  constraints,  the  ‘short’  version  (only  Level  3  for  Conditions,  Processes,  and  Outcomes)  of  the 
original  CTEF  2005  model  was  administered.  Participants  made  four  assessments  for  each  item:  Relevance, 
Magnitude,  Direction  of  impact,  and  Strength  of  impact. 

Of  the  1 1  participants,  10  had  military  experience  (range:  6  -  30  +  years);  four  had  experience  with  effects 
based  planning;  all  had  participated  in  joint  tours  or  postings;  eight  had  participated  in  multi-national 
tours/postings;  and  five  respondents  had  previously  worked  in  a  distributed  collaborative  environment. 
The  background  of  the  eleven  respondents  was  Navy  (2),  Air  Force  (1),  Army  (5),  government  employee  (1), 
and  ex-military  contractor  (2). 

3.2.3  Results  of  Respondent  Feedback 

Many  of  the  participants  found  the  layout  of  the  survey  confusing  and  thus  had  difficulty  with  answering 
each  part  of  the  question.  One  notable  and  consistent  problem  was  with  the  “direction  of  impact”  question. 
For  example,  people  sometimes  answered  ‘0’  to  the  direction  of  impact,  but  then  indicated  a  value  for  the 
strength  of  impact,  which  does  not  seem  consistent.  It  was  clear  that  it  would  be  better  to  employ  two 
rating  scales  for  each  item  instead  of  four.  In  fact,  we  went  back  to  the  original  structure  and  decided  to  use 
one  scale  for  rating  the  magnitude  of  the  item  and  a  second  “strength  of  impact”  rating  scale,  with  each 
employing  a  5-point  Likert  scale. 

Other  feedback  on  the  CTEF-instrument  resulted  from  a  global  perspective  on  team  performance  in  the 
study.  Specifically,  the  type  of  team  in  this  sample  was  not  typical  of  a  command  team.  They  did  not  have 
at  their  disposal  the  tools  they  typically  used  in  operations,  the  EBO  process  was  new  to  them,  and  the 
technology  to  link  the  distributed  teams  was  constantly  going  down,  which  influenced  their  ability  to  get 
their  job  done.  Accordingly,  it  was  clear  that  we  should  add  an  item  to  the  CTEF  reflecting  the  impact  of 
technology  on  team  effectiveness. 

3.2.4  Conclusion 

Based  on  the  results  of  our  experiences  with  MNE4,  we  decided  to  stay  with  the  original  two  scales, 
for  magnitude  and  impact. 


3.3  SUPREME  ALLIED  COMMAND  TRANSFORMATION  (SACT) 

We  were  provided  with  the  opportunity  to  collaborate  with  the  HFM-138  Task  Group  on  Adaptability  in 
Coalition  Teamwork  to  collect  feedback  on  the  CTEF  instrument  from  personnel  at  NATO’s  Supreme 
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Allied  Command  Transformation  (SACT)  Headquarters  in  Norfolk,  VA.  SACT  headquarters  directs 
NATO’s  Allied  Command  Transformation  (ACT)’s  various  subordinate  commands  such  as  the  Joint 
Warfare  Centre  (JWC),  The  Joint  Forces  Training  Centre  (JFTC),  and  various  NATO  Schools. 

3.3.1  Context 

While  we  were  interested  in  applying  the  CTEF  instrument  in  operational  environments,  we  did  not  want 
miss  the  opportunity  to  receive  feedback  on  the  instrument  from  multi-national  military  personnel. 
Therefore,  the  main  goal  of  this  data  collection  effort  was  to  gain  feedback  and  insight  on  how  to  make  the 
instrument  simpler  and  more  usable.  Specifically,  we  wanted  to  gauge  respondent’s  perceptions  of  reading 
comprehension  level,  reduce  confusion  of  item  wording,  and  increase  the  clarity  of  the  level  and  impact 
scales. 


3.3.2  Method 

Twenty -two  participants  (19  male  and  3  female;  13  military  and  9  civilian)  from  SACT  HQ  completed  the 
CTEF  survey.  The  seven  nations  represented  in  this  sample  included:  Canada,  France,  Great  Britain, 
Greece,  the  Netherlands,  Spain,  and  the  United  States.  Two  members  of  the  HFM-138  Adaptability  in 
Coalition  Teamwork  panel  provided  the  medium  length  version  of  the  CTEF  to  participants  in  one  of 
3  ways:  Interview  (N  =  2);  Paper-and-pencil  (N  =  13);  and  Desktop  computer  program  (N  =  7). 

Following  the  CTEF  questionnaire,  participants  were  asked  to  provide  reactions  and  feedback  on  the 
instrument. 

3.3.3  Results  of  Respondent  Feedback 

The  reduced  reading  level  that  was  introduced  after  the  first  administration  in  Joint  Caribbean  Lion 
(see  Section  3.5)  did  increase  the  understanding  of  the  items  overall,  although  there  were  still  a  few  items 
that  needed  re-wording  and  clarification  based  on  the  SACT  personnel  feedback.  For  example,  a  few 
respondents  commented  that  they  were  confused  about  the  difference  between  questions  regarding  the  use 
of  knowledge  (processes)  versus  possessing  knowledge  (conditions).  Additionally,  a  large  majority  of 
participants  did  not  understand  the  item  ‘team  maturity’.  These  comments  led  to  minor  changes  in  the 
wording  of  the  items. 

The  participants  were  asked  to  describe  their  team  following  the  completion  of  the  instrument.  However, 
many  of  the  participants  did  not  understand  which  team  was  being  referred  to:  was  it  the  team  that  they 
were  in  now,  or  the  team  they  were  part  of  during  their  daily  work?  Also,  the  concept  of  team  was  not  very 
clear  to  them. 

Finally,  a  majority  of  the  participants  were  confused  about  the  magnitude  and  the  impact  on  team 
effectiveness  scale  values.  The  major  cause  of  confusion  seemed  to  stem  from  the  participants  rating  items 
on  the  magnitude  scale  ranging  from  1  to  5  and  then  rating  impact  on  a  scale  ranging  from  -2  to  +2. 
Also,  the  SACT  personnel  indicated  that  there  is  a  difference  between  no  impact  on  performance  and 
neutral  impact  on  performance. 

3.3.4  Conclusions 

As  a  result  of  the  feedback,  we  implemented  two  changes  to  the  instrument.  First,  we  moved  the  team 
description  questions  from  a  separate  demographics  list  into  the  CTEF  questionnaire  itself.  Additionally, 
we  updated  the  instructions  to  make  this  connection  to  the  team  more  salient.  For  example,  the  instructions 
now  read  for  the  survey:  “ Questions  on  the  following  pages  will  ask  you  to  reflect  on  the  performance  and 
effectiveness  of  a  command  team  you  have  been  a  part  of  as  a  leader  or  member.  Please  consider  one 
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command  team  that  you  have  been  a  part  of,  in  a  recent  deployment  or  other  operational  context. 
Remember  that  this  team  will  be  the  reference  of  ALL  upcoming  questions 

To  reduce  the  confusion  about  the  scales,  we  added  text  anchors.  For  example,  the  magnitude  scales  were 
labelled  Very  Low,  Low,  Moderate,  High,  and  Very  High.  The  impact  scales  we  labelled  to  read  Very 
Negative,  Negative,  No  Impact,  Positive,  Very  Positive.  Also,  a  “not  applicable”  option  was  added  to  the 
impact  scale. 


3.4  GAMMEL  DANSK  (GD) 

Gammel  Dansk  is  an  international  naval-air  training  exercise  executed  yearly  at  the  Naval  Operational 
School  in  Den  Helder,  Netherlands.  It  was  suggested  by  the  head  observer  of  Joint  Caribbean  Lion  that  the 
evaluation  of  the  exercise  could  be  improved  by  making  use  of  CTEF.  For  us  it  was  another  opportunity  for 
data  sampling.  We  also  wanted  the  respondents  to  evaluate  the  questionnaire. 

3.4.1  Context 

The  Gammel  Dansk  exercise  has  an  educational  purpose,  and  is  a  final  exercise  and  test  at  the  end  of  a  six 
week  course  on  sea-air  integrated  operations.  The  purpose  of  the  course  is  to  train  naval  and  air  defence 
personnel  in  operational  command,  collaboration,  and  teamwork.  Both  trainers  and  trainees  took  part  in 
the  exercise.  The  trainees  were  assigned  to  the  different  teams  in  the  exercise. 

3.4.2  Method 

The  exercise  involved  five  ship  teams;  four  Air-Defence  teams;  and  some  other  respondents.  On  the  last 
day  of  the  exercise  the  medium  version  of  the  CTEF  instrument  was  administered  as  a  paper-and-pencil- 
questionnaire.  It  was  the  same  questionnaire  that  was  used  for  JCL2  and  JCL3,  but  it  was  translated  back 
into  English.  However,  it  was  the  first  time  that  we  added  feedback  loop  items  and  evaluation  items. 
Instructions  to  fill  out  the  questionnaire  were  given  verbally  by  the  Netherlands  Task  Group  representatives. 

3.4.3  Results 

Forty-eight  participants  (44  male  and  4  female)  from  Germany  (N  =  23),  the  Netherlands  (N  =  22), 
Belgium  (N  =  2)  and  Poland  (N  =  1)  completed  the  CTEF  survey. 

Feedback  to  the  commander  was  given  for  each  team.  In  each  case,  we  presented  mean  results  and  compared 
them  with  the  results  of  the  other  teams.  The  comparisons  have  not  been  statistically  tested,  because  the 
number  of  respondents  per  team  was  too  small.  Here,  we  report  only  some  general  trends. 

3.4.3. 1  Conditions 

•  Teams  estimated  their  team  leader  (M  =  3.8)  and  team  members  (M  =  3.6)  as  rather  competent. 

•  Team  (M  =  3.3)  and  Organisation  (M  =  3.2)  were  viewed  somewhat  less  favourably. 

•  Mission  framework  (M  =  2.3)  and  the  task  (M  =  2.6)  generally  did  not  seem  to  be  too  complicated. 

•  One  team  agreed  on  the  negative  impact  of  a  number  of  conditions  for  their  team  effectiveness, 
especially  Unclear  goals,  Team  leader  skills,  Adequate  mix  of  people,  and  Team  maturity.  Other  teams 
were  very  positive  about  the  impact  that  their  team  leader  and  team  members  had  on  the  effectiveness  of 
their  team. 
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3.43.2  Processes 

•  Teams  scored  well  on  all  Task-focused  behaviours  (M  =  3.6). 

•  Teams  scored  well  on  all  Team-focused  behaviours  (M  =  3.7),  except  for  Motivating  (M  =  3.3). 

•  Most  teams  agreed  that  their  level  of  Task-focused  and  Team-focused  behaviours  had  a  positive 
impact  on  team  effectiveness. 

3.4.3.3  Outcomes  and  Feedback 

•  Teams  scored  very  well  on  task  outcomes  (M  =  3.6)  and  team  outcomes  (M  =  4.0)  and  task  feedback 
(M  =  4.0)  and  team  feedback  (M  =  3.8). 

3.4.3.4  Evaluation  of  the  Items 

The  results  of  the  evaluation  questions  are  described  In  Table  3-1. 


Table  3-1:  Seven  Evaluation  Items  (Percentage;  N  =  48). 


(Completely) 

Disagree 

Neutral 

(Completely) 

Agree 

I  think  that  the  right  aspects  of  team 
effectiveness  are  being  addressed 

7 

40 

53 

I  think  the  questions  are  clear 

7 

40 

53 

I  think  that  the  distinction  between  level  and 
impact  is  clear 

2 

31 

67 

I  think  that  the  distinction  between  level  and 
impact  is  useful 

5 

34 

61 

I  think  that  the  length  of  the  questionnaire  is 
about  right 

26 

44 

30 

I  learned  something  about  improving  team 
effectiveness 

32 

34 

32 

I  would  want  to  use  the  instrument  in  practice 

37 

30 

33 

3.4.4  Conclusions  of  Gammel  Dansk 

Gammel  Dansk  revealed  that  respondents  were  very  positive  about  their  team  effectiveness.  Especially, 
the  team  leader  and  team  members  were  evaluated  very  positively.  In  general,  the  respondents  were  also 
positive  about  the  model  and  the  instrument.  Most  of  the  respondents  thought  that  the  right  aspects  of  team 
effectiveness  were  being  addressed  and  that  the  questions  were  clear.  Furthermore,  they  found  the  distinction 
between  the  level  and  impact  scores  both  clear  and  useful.  However,  only  one  third  of  the  respondents  found 
the  length  of  the  questionnaire  right  and  also  only  one  third  would  like  to  use  the  instrument  in  practice. 


3.5  JOINT  CARIBBEAN  LION 

Joint  Caribbean  Lion  was  a  joint,  combined,  amphibious  (sea-land)  exercise  led  by  the  Netherlands, 
including  troops  from  the  United  States  of  America,  United  Kingdom,  Canada,  France,  and  Belgium. 


3-6 


RTO-TR-HFM-127 


DEVELOPMENT  AND  INITIAL  APPLICATIONS  OF  THE  INSTRUMENT 


The  opportunity  to  issue  the  CTEF-instrument  came  up  during  meetings  between  TNO  and  the  Royal 
Netherlands  Navy.  The  commanders  of  the  Dutch  maritime  contingent  wanted  to  evaluate  their  staff 
during  the  exercise  and  improve  it  systematically  over  time.  They  had  seen  the  CTEF  model  and 
recognized  its  potential  for  their  purpose.  The  implementation  of  the  CTEF-instrument  for  this  exercise  is 
described  in  Essens  et  al.  (2008). 

3.5.1  Context 

As  stated  on  the  website  (Dutch  Ministry  of  Defence,  2007),  the  “Main  purpose  of  the  exercise  Joint 
Caribbean  Lion”  was  “to  mobilize  an  expeditionary  military  force  of  the  Dutch  defence  forces,  supplemented 
with  international  participants,  in  order  to  react  to  a  crisis  in  an  out-of-area  environment”. 

The  joint  and  combined  (all  services  and  multi-national)  exercise  was  conducted  in  the  period  from 
15  May  to  16  June,  2006  -  the  first  ever  Netherlands  Joint  (multi-service)  and  Combined  (multi-national) 
exercise  on  such  a  large  scale.  Besides  the  above  mentioned  troop  contributing  nations,  Venezuela  and 
Chile  contributed  with  observers.  About  4500  military  personnel  participated  with  12  ships,  2  maritime 
patrol  aircrafts,  four  F-16s,  and  transport  planes.  The  exercise  was  based  upon  Chapter  VII  UN  operation. 

The  Dutch  maritime  contingent  consisted  of  a  Command  Amphibious  Task  Force  and  a  Command 
Landing  Force.  The  two  staffs  merged  into  one  joint  staff  to  plan  and  execute  the  maritime  and 
amphibious  operations.  The  command  and  control  organization  was  new  and  exercising  for  the  first  time. 
Several  positions  were  filled  shortly  before  the  exercise. 

The  commander’s  intent  was  that  the  whole  staff  (about  70)  would  fill  in  an  electronic  version  of  the 
CTEF  instrument  at  several  points  during  the  exercise,  and  that  they  would  electronically  send  the  data 
files  to  the  members  of  the  HFM-127  Task  Group  to  be  processed  at  short  notice  for  the  Commander’s 
operational  feedback  two  days  later.  Data  collection  at  Joint  Caribbean  Lion  was  the  first  large  data 
collection  opportunity.  Most  of  the  procedures  of  this  distant  data  collection,  data  processing, 
and  operational  feedback  had  to  be  developed  from  scratch.  We  wanted  to  determine  if  teams  could  work 
with  the  model  and  the  instrument,  if  the  instrument  worked  as  a  tool  for  team  improvement,  and  if 
different  kinds  of  team  members  (i.e.,  leaders  and  members)  would  fill  in  the  items  in  the  same  way. 
Furthermore,  we  wanted  to  examine  in  more  detail  the  relationship  between  the  magnitude  and  the  impact 
scores. 

During  the  Joint  Caribbean  Lion  exercise  a  tropical  storm  passed  through  the  area  of  operations  that 
caused  a  major  flood  in  Surinam.  As  a  result,  the  population  of  Surinam  was  in  need  of  emergency 
support,  which  was  provided  by,  amongst  others,  the  crew  and  staff  involved  in  the  Dutch  contingent  of 
Joint  Caribbean  Lion. 

3.5.2  Method 

The  participants  were  leaders  of  a  command  team  or  a  staff  team  (both  navy  and  army  teams),  members  of 
those  teams  (officers  and  NCOs),  or  observers  of  the  exercise.  A  stand-alone  desktop  version  of  the  CTEF 
instrument  was  used  and  made  available  to  the  staff  via  the  ship’s  network.  The  administration  of  the 
survey  was  overseen  by  the  head  of  the  observers.  The  instrument  was  filled  out  at  the  beginning  of  the 
exercise  (Tl),  halfway  (T2),  and  at  the  end  of  the  exercise  (T3).  The  respondents  filled  out  the 
questionnaire  at  selected  computer  terminals  within  two  days  after  the  leader  of  the  observers  requested 
them  to  do  so.  He  collected  the  raw  data  files  and  sent  them  to  us  via  email.  Within  48  hours  we  processed 
and  analysed  the  data  and  made  a  concept  (PowerPoint)  presentation,  which  could  be  adapted  by  the  lead 
observer  to  his  own  needs  for  giving  feedback  to  the  staff  of  JCL.  The  results  were  presented  and 
discussed  with  the  personnel  that  were  present.  At  Tl  not  all  personnel  were  in  place  and  therefore  not  all 
were  able  to  fill  out  the  questionnaire  at  this  point  in  time. 
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In  Table  3-2  an  overview  is  given  of  the  number  of  responses  per  category  for  each  point  in  time. 


Table  3-2:  The  Number  of  Respondents  at  Time  T1,  T2,  T3. 


Tl 

T2 

T3 

Team  Leaders  (and  deputy  team  leaders) 

10 

14 

27 

Team  Members 

19 

40 

28 

Observers 

4 

4 

4 

Incomplete  Questionnaires 

7 

5 

5 

Total  Valid  Questionnaires 

33 

58 

59 

3.5.3  Results 

After  Tl,  the  first  JCL  administration,  the  lead  observers  reported  that  the  Dutch  staff  had  serious 
complaints  about  the  readability  and  the  length  of  the  instrument.  This  led  us  to  change  the  instrument  to  a 
new  version  which  was  used  at  T2  and  T3.  We  will  discuss  separately  Tl  and  T2  +T3  results,  referred  to 
as  JCL1  and  JCL  2  and  JCL3  below. 

3. 5.3.1  Results  of  First  Administration  -  JCL  1 

Forty  participants  from  the  command  ship  HMS  Rotterdam  responded  at  JCL  1.  Of  those,  7  were  incomplete 
or  duplicates,  hence,  33  responses  were  used  for  analysis.  Of  these,  10  were  leaders,  19  were  team  members 
(4  N/G2,  4N/G/SA3,  10  N/G5,  1  N/G6),  and  4  were  observers  on  the  ship. 

Issues  that  came  to  our  attention  from  this  phase: 

•  The  concept  of  team  did  not  seem  to  be  clear  in  this  context.  Are  the  different  staff  sections  the 
teams?  Did  some  questions  refer  to  the  staff  as  a  whole?  For  the  lead  observer,  the  concept  of 
team  that  we  as  researchers  had  in  mind,  namely  a  commander  plus  his  sub-commanders  or 
section  heads,  was  not  clear.  There  were  many  ‘teams’  within  the  staff  and  one  person  could  work 
in  more  than  one  team.  Therefore,  we  needed  to  be  clear  which  team  we  were  interested  in. 

•  Some  respondents  indicated  a  concern  about  privacy,  because  we  asked  them  to  identify  in  what 
section  they  were  in,  and  whether  they  were  a  leader  or  a  team  member.  Was  the  commander  or 
the  leader  of  the  observers  reading  their  responses?  Could  they  give  candid  responses  without 
being  identified,  in  particular  in  small  sections?  Guarantees  were  given  that  the  data  would  be 
handled  confidentially  and  that  the  results  would  only  be  presented  at  group  level. 

•  The  wording  of  the  instrument  was  considered  to  be  too  difficult  for  the  respondents.  Further 
analyses  revealed  that  the  reading  level  that  was  needed  to  read  those  items  was  indeed  too  high 
(13th  grade). 

•  The  respondents  had  difficulty  answering  the  English  items.  They  preferred  the  questionnaire  to 
be  translated  into  Dutch. 

•  The  time  needed  to  answer  the  instrument  was  too  long.  The  number  of  questions  and  the  time 
each  question  took  needed  to  be  reduced. 

3. 5.3.2  Conclusion  JCL  1 

As  a  result  of  the  aforementioned  comments  we  decided  to  use  the  medium  version  of  the  instrument  for 
the  next  measurements  (This  version  became  the  “standard”  version  for  subsequent  measurements). 
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Furthermore,  we  decided  to  optimise  the  readability  level  of  the  items  from  1 3th  grade  to  9th  grade  level 
(using  the  Flesch-Kincaid  Level  for  readability),  and  also  translated  the  items  into  Dutch  for  the 
respondents  of  Joint  Caribbean  Lion  (administrations  JCL  2  and  JLC  3).  With  respect  to  the  concept  of 
team,  we  analysed  the  data  at  the  general  level  and  did  not  split  the  results  for  the  different  sub-teams  that 
could  have  been  distinguished. 

3. 5.3.3  Results  of  Second  and  Third  Administration  -  JCL  2  +  3 

Most  respondents  reported  which  team  they  belonged  to,  but  we  did  not  report  back  the  results  per  team  to 
avoid  identification  of  individuals.  First,  it  was  not  clear  which  team  they  were  actually  referring  to  in 
answering  the  questions:  were  they  referring  to  their  section  or  to  the  larger  staff?  And  second,  the  number 
of  respondents  per  team  was  rather  small.  Therefore,  in  the  remainder  of  this  section  only  aggregated 
results  will  be  reported. 

We  computed  the  overall  correlation  between  the  magnitude  and  the  impact  scores  for  each  item  at  Tl,  T2, 
and  T3.  The  mean  results  were  .57  (Tl),  .77  (T2),  .78  (T3).  These  correlations  are  very  high  with  the 
exception  of  Mission  Framework  items  and  Task  items  (for  Tl  .18  and  .13  respectively,  all  others  above 
.60;  for  T2  .36  and  .35  respectively,  all  others  .68  or  above;  for  T3  -.02  and  .22  respectively,  all  others 
above  .82).  Because  of  the  high  correlations  for  most  of  the  items,  we  chose  not  to  give  feedback  on  both 
magnitude  and  impact. 

We  chose  to  present  the  impact  scores,  which  represented  the  respondents’  perception  of  the  effect  of  a 
particular  level  of  a  condition  or  process  on  overall  effectiveness.  Since  there  is  no  norm  on  the  meaning  of 
the  magnitude  or  level  of  an  aspect,  the  impact  question  better  taps  into  the  respondent’s  model  of  what  is 
good  and  what  should  be  improved.  Furthermore,  team  members  might  be  more  convinced  to  change  an 
aspect  of  their  team’s  functioning  if  they  were  given  information  that  a  relatively  large  number  of  the  team 
members  thought  that  team  effectiveness  was  negatively  impacted  by  that  aspect.  Therefore,  the  core  of  the 
presentations  that  were  prepared  consisted  of  an  analysis  of  positive  or  negative  impact  scores. 
For  each  component  of  the  questionnaire  we  presented  those  items  for  which  at  least  30%  of  the  respondents 
reported  (highly)  negative  or  (highly)  positive  impact  scores.  By  presenting  this,  the  participants  could  easily 
see  the  perceived  weak  or  strong  points  of  the  whole  team.  Figure  3-2  gives  an  example  of  a  slide  that  was 
presented  in  the  Commander  and  Staff  review. 


CONDITIONS 


Mission 

framework 


Organisation 


Team  member 


Conditions:  Organisation 


Items  with  most 
negative  impact: 

Clarity  of  command 
structure  (40%) 


Joint  Caribbean  Lion  2nd  Assessment 


Items  with  most 
positive  impact: 

Congruity  of  team’s 
mission  and  org. 
goals  (62%) 
Autonomy  (53%) 
Organisational 
support  (48%) 


Figure  3-2:  Sample  of  a  Slide  in  the  Feedback  Presentations  (from  T2  Assessment). 
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In  Table  3-3  we  summarize  which  items  show  consistent  negative  or  consistent  positive  impact  scores  on 
team  effectiveness.  That  is,  at  all  points  in  time  (Tl,  T2,  and  T3)  more  than  30%  of  the  respondents 
indicated  that  the  item  had  a  negative  or  a  positive  impact  on  team  effectiveness.  Table  3-3  shows  that  the 
respondents  consistently  considered  their  situation  as  being  so  uncertain  and  their  task  having  such  a  high 
workload  that  it  negatively  impacted  effectiveness.  It  also  shows  that  they  considered  their  teams  as 
having  the  right  people,  but  they  were  negative  about  the  maturity  of  the  team  as  a  whole.  With  respect  to 
the  processes,  they  saw  information  exchange  as  a  problem,  but  decision  making  and  backing-up  as  a 
strength.  Finally,  they  considered  the  achievement  of  task  goals  of  several  stakeholders  and  the  mutual 
trust  and  confidence  as  strengths. 


Table  3-3:  Items  Rated  as  Either  a  Negative  (-)  or  Positive  (+)  Impact  on 
Team  Effectiveness  Consistent  Over  Tl,  T2,  and  T3  (>30%  Criterion). 


Conditions 

Mission 

Framework 

Situation  uncertainty  (-) 

Leader 

Skills  (+) 

Knowledge  (+) 

Match  personal  to 
organisational  goals  (+) 

Task 

Workload  (physical)  (-) 
Unstable  goals  (-) 

Team  Member 

Skills  (+) 

Knowledge  (+) 

Organisation 

Autonomy  (+) 

Team 

Composition  (+) 

Maturity  (-) 

Processes 

Task-Focused 

Behaviours 

Managing  information 
(exchanging)  (-) 

Making  decisions  (+) 

Team-Focused 

Behaviours 

Adapting  to  changes 
(backing  up)  (+) 

Outcomes 

( for  intermediate  or  end  goals) 


Task 

Goal  achievement  (+) 

Team 

Mutual  trust  (+) 

Outcomes 

Stakeholders’  expectations 

Outcomes 

Morale  (+) 

(+) 

Collective  confidence  in 
success  (+) 

Mutual  respect  (+) 

In  order  to  see  if  there  was  improvement  in  the  impact-scores  over  time,  we  computed  the  percentage  of 
positive  and  negative  impact  scores  for  all  items  within  each  component  of  the  model  (see  Table  3-4). 
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Table  3-4:  Trends  in  Impact  Scores  During  the  Exercise 
(Underlined  the  Highest  Score  of  the  Three  Administrations). 


Number  of  Impact  Scores  (%) 
Negative  Scores  Positive  Scores 


Tl 

T2 

T3 

Tl 

T2 

T3 

Mission  Framework 

28 

34 

30 

19 

11 

17 

Task 

43 

48 

41 

20 

10 

17 

Organisation 

36 

26 

27 

35 

47 

39 

Leader 

20 

14 

18 

44 

50 

48 

Team  Members 

21 

19 

20 

48 

51 

44 

Team 

26 

32 

44 

37 

37 

28 

Task-Focused  Behaviours 

30 

22 

32 

34 

35 

25 

Team-Focused  Behaviours 

25 

22 

11 

30 

36 

25 

Task  Outcomes 

18 

9 

8 

35 

38 

44 

Team  Outcomes 

23 

24 

24 

43 

37 

33 

Table  3-4  shows  that  the  mission  framework  and  the  task  had  the  worst  consequences  for  team 
effectiveness  at  T2.  For  the  task  items  almost  half  of  the  respondents  (48%)  estimated  a  negative  impact  at 
T2.  At  that  point  in  time  the  emergency  support  in  Surinam  had  its  most  significant  impact  on  the  exercise. 
On  the  other  hand,  the  respondents  thought  that  their  organisation,  their  team  leaders,  and  team  members 
fulfilled  the  most  positive  role  at  that  same  point  in  time.  At  that  point  in  time  also  the  task-focused  and 
team-focused  behaviours  played  the  most  positive  role  for  team  effectiveness.  A  surprising  effect  was  that 
the  team,  as  well  as  the  task-focused  and  team-focused  behaviours,  had  their  most  negative  effects  at  T3, 
the  point  in  time  where  the  team  should  have  been  the  best,  because  of  the  growth  as  a  team  as  a  result  of 
the  exercise. 

3.5.4  The  Magnitude  Scores 

A  separate  analysis  was  done  on  the  magnitude  scores.  We  compared  the  differences  between  the  three 
categories  of  respondents  over  time:  leaders,  team  members,  and  observers  at  Tl,  T2,  and  T3. 
We  computed  the  mean  scores  by  averaging  the  responses  to  all  items  that  belong  to  a  certain  component. 
Table  3-5  shows  that  in  general  the  leaders  gave  the  most  positive  scores,  then  the  team  members, 
and  followed  by  the  observers.  The  observers  scored  much  lower  than  the  other  respondents  on  team 
members,  team,  task-focused  behaviours,  team-focused  behaviours,  task  outcomes,  and  team  outcomes. 
In  other  words,  the  observers  assessed  these  components  as  more  negative  than  the  team  members  and 
leaders.  The  observers  also  scored  the  mission  framework  lower,  which  means  that  they  perceived, 
e.g.,  less  uncertainty  in  the  mission  than  the  teams  themselves  (as  an  example).  Although  they  were  rather 
negative  in  their  assessments  at  the  start  of  the  exercise,  the  observers  grew  more  positive  in  their 
assessments  of  task-focused  behaviours,  team-focused  behaviours,  and  task  outcomes  during  the  exercise, 
whereas  they  became  more  negative  about  the  organisation  and  the  team  outcomes.  Contrary  to  the 
prevailing  positive  trends  of  the  observers,  is  the  negative  trend  of  the  team  leaders  with  respect  to  the 
team.  The  team  leaders  perceived  the  team  to  have  less  positive  characteristics  at  the  end  of  the  exercise 
than  at  the  beginning. 
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Table  3-5:  The  Mean  Scores  of  the  Items  per  Component  for  Different  Respondent  Categories 
(Bolded  if  the  differences  between  the  respondent  categories  are  larger  than  0.4; 
Underlined  if  the  respondents  changed  their  assessments  substantially  (>0.4)  over  time). 


Average  Scores  (Scale  1  -  5) 


T1 

T2 

T3 

L*) 

M 

O 

L 

M 

O 

L 

M 

O 

Mission  Framework 

3.54 

2.80 

2.56 

3.06 

2.82 

2.31 

3.07 

3.03 

2.38 

Task 

3.05 

3.10 

3.07 

3.22 

2.43 

3.06 

3.28 

3.39 

3.06 

Organisation 

3.32 

2.96 

3.25 

3.56 

3.23 

3.44 

3.15 

3.10 

2.69 

Leader 

3.53 

3.25 

3.58 

3.54 

3.36 

3.67 

3.37 

3.38 

3.58 

Team  Members 

3.52 

3.42 

2.92 

3.44 

3.36 

3.08 

3.16 

3.25 

3.25 

Team 

3.33 

3.12 

2.90 

2.95 

3.13 

2.80 

2.67 

2.90 

2.90 

Task-Focused  Behaviours 

3.34 

3.06 

2.61 

3.32 

3.08 

2.74 

2.92 

2.82 

3.16 

Team-Focused  Behaviours 

3.18 

2.90 

2.49 

3.06 

3.14 

2.92 

2.84 

2.84 

2.86 

Task  Outcomes 

3.64 

3.11 

2.83 

3.29 

3.34 

2.92 

3.47 

3.28 

3.56 

Team  Outcomes 

3.27 

3.34 

3.00 

3.09 

3.29 

2.50 

3.04 

3.12 

2.61 

*)  L=  Team  Leader;  M  =  Team  Member;  O  =  Observer. 


3.5.5  Usability 

The  CTEF  model  was  recognized  by  the  commanders  as  a  relevant  set  of  items  that  should  be  addressed. 
The  systematic  review  of  factors  enabled  by  the  CTEF  model  and  the  results  from  the  assessments  were 
highly  appreciated  by  the  command.  The  resulted  provided  a  basis  for  discussion  with  the  crew  on  what 
the  positive  and  negative  issues  were  and  what  actions  should  be  taken  to  repair  the  negative  trends. 
As  mentioned  previously,  in  particular  at  the  first  administration  the  instrument  was  seen  by  the  crew  as 
too  complex  in  wording  and  it  was  too  long  for  operational  conditions.  Most  agreed  that  the  systematic 
review  was  supportive,  but  that  it  could  also  lead  to  too  long  review  sessions.  A  subtle  point  was  that  some 
had  a  problem  in  revealing  their  judgments,  despite  the  fact  that  data  were  collected  anonymously. 

Another,  complicating  but  interesting,  issue  was  that  the  definition  of  ‘which  team  do  you  belong  to’  was 
not  simple  to  answer  for  many  of  the  crew,  because  they  considered  themselves  often  part  of  more  than 
one  team.  And  also,  a  team  leader  is  mostly  a  member  of  a  higher  level  team. 

3.5.6  Conclusion  and  Discussion 

The  main  objectives  of  the  Joint  Caribbean  Lion  application  were  to  support  the  commanders  to  have 
better  team  reviews  and  improve  team  effectiveness.  In  particular:  Do  the  CTEF  model  and  the  instrument 
work  for  commanders  and  the  teams  to  address  team  effectiveness,  and  does  that  lead  to  team 
improvement?  Feedback  from  the  commanders  revealed  that  the  model  was  immediately  recognized  as 
militarily  relevant  and  that  the  structure  of  the  model  helped  to  systematically  address  issues  in  discussing 
their  staffs  performance. 
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One  of  the  questions  about  the  instrument  that  was  addressed  was  whether  the  measurement  of  both 
magnitude  and  impact,  which  both  seemed  relevant,  is  necessary.  A  calculation  of  the  correlations  between 
magnitude  and  impact  showed  that  they  were  almost  all  very  high.  This  implies  that  either  both  scales  seem 
to  measure  the  same  underlying  construct  or  respondents  were  not  able  to  distinguish  between  the  two. 

Application  of  the  instrument  in  a  process  of  assessment,  reporting  and  discussion  during  team  review 
sessions  should  lead  to  improved  team  effectiveness.  During  the  exercise  Joint  Caribbean  Lion  the  results 
did  not  show  improved  in  team  effectiveness.  Only  the  observers  saw  some  positive  trends  in  team 
performance.  At  least  three  questions  may  be  asked  about  this  failure  to  show  improvement.  First,  were 
the  measures  reliable?  Perhaps,  the  team  functioning  may  have  improved,  but  the  instrument  was  not  able 
to  measure  it.  In  order  to  check  this  possible  conclusion,  we  had  a  final  evaluation  with  the  commander  of 
the  exercise  and  the  leader  of  the  observers.  They  both  confirmed  the  observation  from  the  questionnaires 
that  the  teamwork  had  not  improved  during  the  exercise. 

Second,  did  the  relief  operation  in  Surinam,  which  broke  up  the  organisation,  have  a  negative  effect  on 
team  improvement?  Especially  at  T2  this  should  have  had  effects  on  the  data,  because  at  that  point  in  time, 
a  number  of  the  team  members  had  been  extracted  from  the  exercise.  The  data  show  that  at  that  point  in 
time,  especially  the  mission  framework  and  the  task  had  the  most  negative  impact  scores.  That  can  be 
explained  because  the  situation  created  extra  uncertainty  and  pressure  on  the  team.  On  the  other  hand, 
at  that  point  in  time  a  lot  of  other  aspects  showed  their  most  positive  impact  scores.  For  example, 
the  organisation,  leaders,  team  members,  and  processes  were  perceived  to  have  relatively  high  positive 
impact  on  team  effectiveness.  This  may  be  explained  by  the  positive  feelings  that  were  generated  doing 
something  to  relieve  people  in  Surinam  from  their  problems.  People  may  have  felt  they  were  doing  a  good 
job  and  working  hard  to  solve  the  problems. 

Third,  were  the  feedback  review  sessions  that  were  organized  the  best  way  to  work  on  the  improvement  of 
the  teamwork?  In  those  meetings,  the  data  were  presented  and  discussed  with  all  people  that  were  present 
at  the  location  (i.e.,  one  of  the  ships).  With  such  a  large  ‘team’  together  and  still  a  number  of  people  not 
present,  it  may  not  have  been  clear  who  was  responsible  for  dealing  with  the  problems  in  team  functioning 
that  came  up  during  the  presentation.  Furthermore,  in  hindsight  it  may  not  have  been  the  right  aggregation 
level  to  feed  back  the  data.  Perhaps,  the  teams  that  should  have  received  specific  feedback  were  the 
smaller  teams  that  composed  the  larger  team  (e.g.,  the  operations  team,  the  intel  team).  The  feedback 
should  perhaps  have  been  targeted  at  their  level.  On  the  other  hand,  involvement  of  the  whole  staff  may 
raise  shared  awareness  of  the  state  and  direction  of  effectiveness  of  the  staff. 

The  question  of  who  should  fill  out  the  instrument  is  still  open.  CTEF  was  intended  to  be  used  by  command 
teams  (commander  -  staff/section  heads  or  commander  -  sub-commanders).  The  commander  and  the  leader 
of  the  observers  wanted  to  use  it  for  the  whole  staff  at  operational  functions.  The  results  show  that  if  the 
leader  uses  the  instrument  to  assess  own  team  performance,  the  results  may  be  more  positive  than  when  the 
team  members  or  observers  fill  out  the  questions.  When  observers  fill  out  the  questionnaire,  the  problem  may 
be  that  the  team  members  or  team  leaders  may  find  them  to  be  too  disapproving  of  the  team.  At  least  one 
should  be  aware  that  different  tendencies  emerge  if  specific  roles  use  the  instrument.  Limiting  CTEF  to  be 
used  in  a  staff  only  by  the  commander,  chief  of  staff,  and  section  heads  could  solve  the  issue  of  team 
definition  and  might  generate  more  specific  data.  On  the  other  hand,  the  power  of  having  an  instrument  that 
spans  the  whole  operational  staff,  with  multiple  teams  and  dynamic  sub-groups,  is  appealing.  This  might 
require  an  additional  development  of  CTEF  which  is  beyond  our  objectives. 


3.6  CONCLUSIONS 

In  this  chapter  we  described  the  initial  experiences  with  the  instrument  and  a  series  of  administrations  of 
the  instrument  including  a  realistic  operational  exercise  context  (JCL).  The  results  were  used  to  learn  from 
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defence  applications  of  the  CTEF  model  and  instrument.  From  this  a  number  of  conclusions  have  been 
drawn. 

Based  on  the  respondents’  comments  that  the  original  CTEF  version  was  too  long,  a  medium-length 
version  of  the  questionnaire  was  developed  and  has  become  the  standard.  Furthermore,  the  wording  of  the 
items  was  simplified  to  (US)  grade  nine  level  down  from  the  original  grade  thirteen  level.  The  resulting 
items  all  had  a  scale  that  measured  the  magnitude  and  a  scale  that  measured  the  impact  of  that  magnitude. 

The  JCF  longitudinal  administration  of  the  questionnaire  showed  that  the  correlations  between  magnitude 
and  impact  were  high,  except  for  the  aspects  in  the  mission  framework  and  the  task.  This  suggests  that  for 
a  large  part,  magnitude  and  impact  are  measuring  the  same  concept.  Interestingly,  the  items  were  scored 
differently  by  team  leaders,  team  members,  and  observers  -  the  observers  being  most  critical  -  suggesting 
that  the  role  affects  the  perceptions  of  magnitude  and  impact.  Finally,  most  respondents  seemed  to  be 
rather  positive  about  the  model  and  the  instrument,  apart  from  the  length  of  the  latter. 
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Chapter  4  -  MULTI-NATIONAL  CTEF  SURVEY  STUDY 

4.1  INTRODUCTION 

As  discussed  previously,  NATO  RTO  HFM-087  developed  a  comprehensive  theoretical  framework  of 
Command  Team  Effectiveness  (CTEF)  and  an  accompanying  measurement  instrument  that  can  be  applied  in 
training  and  operational  conditions.  Following  on  the  applications  of  the  CTEF  reviewed  in  Chapter  3,  it  was 
evident  that  two  essential  questions  needed  to  be  addressed  by  a  more  extensive  data  collection  opportunity. 
First,  we  needed  to  provide  empirical  verification  of  the  CTEF  theoretical  framework  and  instrument  in 
operational  settings.  Second,  we  needed  to  confirm  that  CTEF  provides  an  effective  method  of  data 
collection  that  is  also  salient  and  usable  for  operational  commanders.  Accordingly,  we  concluded  that  a 
timely  and  important  study  to  undertake  was  an  extensive,  multi-national  survey  study  of  commanders  from 
various  NATO  countries  in  which  the  CTEF  instrument  is  used  as  a  method  of  capturing  their  experiences  as 
leaders  and  members  of  command  teams  in  deployed  operations. 


4.2  OBJECTIVES  OF  THE  SURVEY  STUDY 

The  survey  study  sought  to  provide  an  empirical  verification  of  the  CTEF  model  and  instrument  through 
extensive  data  collection  involving  military  command  team  experiences  in  operations.  The  primary  goals 
of  the  study  included: 

1)  To  examine  the  conceptual  consistency  of  the  model,  model  fit,  and  the  psychometric  properties 
of  the  instrument; 

2)  To  determine  the  applicability  of  the  model  and  the  instrument  for  command  teams  in  different 
NATO  nations  and  diverse  operational  settings;  and 

3)  To  revise,  consolidate,  and  update  the  model  and  instrument  based  on  the  findings  of  the  study. 

Objectives  1  and  2  will  be  the  focus  of  the  current  chapter.  Objective  3  will  be  addressed  in  Chapter  5  and 
include  comparative  analyses  from  previous  applications  of  the  instrument. 

4.2.1  Model  Verification 

The  first  analysis  to  be  conducted  concerned  a  fundamental  distinction  between  judgments  of  magnitude  and 
judgments  of  impact.  Since  the  early  development  of  CTEF  we  have  emphasized  that  the  assessment  of  the 
magnitude  or  level  of  a  particular  item  need  not  be  correlated  with  assessments  of  impact.  For  example, 
high  or  low  levels  of  situational  uncertainty  need  not  correspond  with  respectively  positive  or  negative 
impact  on  the  levels  of  the  task  or  team  outcomes.  This  is  precisely  why  throughout  our  work  with  CTEF  we 
have  collected  data  on  both  magnitude1  assessments  and  impact  assessments.  However,  the  results  of  Joint 
Caribbean  Lion  discussed  in  Chapter  3,  together  with  more  recent  data  collection  opportunities  using  other 
versions  of  the  CTEF  instrument  not  reported  here  (e.g.,  NATO  rapid  response  force  (NRF)  exercise; 
Canadian  Optimized  Battle  Group  (OBG)  exercises),  have  shown  consistently  very  high  correlations 
between  magnitude  and  impact  assessments.  Hence,  despite  the  conceptual  distinction  between  the  two 
measures,  perhaps  in  practice  the  requirement  of  impact  assessments  is  unnecessary.  As  the  assessments  of 
task  and  team  outcome  are  assessed  only  as  magnitudes,  it  would  be  logical  to  preserve  magnitude  scores 
throughout.  In  sum,  the  first  analysis  that  was  conducted  examined  the  correlations  between  assessments  of 
magnitude  and  assessments  of  impact  across  all  CTEF  items.  If  the  correlations  are  found  to  be  significantly 
high  throughout,  then  we  can  limit  follow-on  hypothesis  tests  in  the  present  study  to  assessments  of 


1  Throughout  the  text  of  this  report  we  use  the  terms  magnitude,  level  and  quantity  interchangeably  to  indicate  the  assessment 
of  status  on  a  given  construct. 
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magnitude  only.  More  generally,  statistically  high  correlations  between  magnitude  and  impact  assessments 
would  permit  the  elimination  of  impact  assessments  in  future  versions  of  the  CTEF  instrument. 

Assessment  of  the  conceptual  consistency  of  the  CTEF  model  involved  examination  of  a  number  of 
fundamental  hypotheses  (H„)  concerning  the  major  structural  variables  and  relationships  that  comprise 
command  team  effectiveness.  Hypotheses  1-7  systematically  examine  relational  component  assumptions  of 
the  model.  Hypothesis  8  examines  the  central  theoretical  assumption  of  a  mediational  role  of  the  task-  and 
team-focused  behaviours,  i.e.,  that  the  effects  of  the  conditions  on  task  and  team  outcomes  are  mediated  by 
the  task-  and  team- focused  behaviours.  Hypotheses  9-12  examine  the  role  of  feedback  processes  on  task- 
and  team-focused  behaviours  and  outcomes.  A  summary  of  all  a  priori  hypotheses  is  provided  in  Table  4-1. 


Table  4-1 :  Summary  of  All  a  priori  Hypotheses. 


Hypothesis 

Analysis  Plan 

Formal  Hypothesis 

Hi 

fLxx  -  Ltap  >  0;  pLxx  -  Ltep  >  0 

{xx  =  mf,  ta,  org,  ldr,  mbr,  tm} 

Levels  of  the  condition  factors  will  be  correlated  with 
the  level  of  task-focused  behaviours  and  the  level  of 
the  team-focused  behaviours 

h2 

1  Ltap  -  Ltao  ^  0 

There  will  be  a  positive  relationship  between  the  level 
of  task-focused  behaviours  and  the  level  of  task 

outcomes 

h3 

l*Ltep  -  Lteo  •'>  0 

There  will  be  a  positive  relationship  between  the  level 
of  team-focused  behaviours  and  the  level  of  team 

outcomes 

h4 

I*Ltap  -  Lteo  ^  0 

There  will  be  a  positive  relationship  between  the  level 
of  task-focused  behaviours  and  level  of  team 
outcomes 

h5 

1  Ltep  -  Ltao  ^  0 

There  will  be  a  positive  relationship  between  the  level 
of  team-focused  behaviours  and  level  of  task 

outcomes 

h6 

1  Ltap  -  Ltao(^~)  I" Ltep  -  Ltao(^~) 

Task-  and  team-focused  behaviours  will  be  positively 
correlated  with  task  outcomes,  but  task-focused 
behaviours  will  be  more  highly  correlated 

h7 

l*Ltep  -  Lteo(^~)  '>  J*Ltap  -  Lteo(~^) 

Task-  and  team-focused  behaviours  will  be  positively 
correlated  with  team  outcomes,  but  team  processes 
will  be  more  highly  correlated 

h8 

Conditions  -)  Processes  -) 
Outcomes 

The  relationships  between  conditions  and  task  and 
team  outcomes  will  be  mediated  by  task-  and  team- 
focused  behaviours 

h9 

Ffdbk  -  Ltao  ^  0 

Teams  that  used  feedback  processes  will  have  higher 
task  outcomes  than  teams  that  did  not  use  feedback 
processes 

Hio 

Ffdbk  -  Lteo  '>  0 

Teams  that  used  feedback  processes  will  have  higher 
team  outcomes  than  teams  that  did  not  use  feedback 
processes 

H„ 

Tfdbk  -  Ltap  ^  0 

Teams  that  used  feedback  processes  will  have  higher 
levels  of  task-focused  behaviours  than  teams  that  did 
not  use  feedback  processes 

Hl2 

1  fdbk  -  Ltep  ^  0 

Teams  that  used  feedback  processes  will  have  higher 
levels  of  team-focused  behaviours  than  teams  that  did 
not  use  feedback  processes 
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The  first  hypothesis  sought  to  examine  the  relationship  between  the  level  of  the  condition  factors  and  the 
level  of  the  task-  and  team-focused  behaviours.  We  therefore  hypothesized  the  following: 

•  Hp  Condition  factors  will  be  correlated  with  task- focused  behaviours  and  team-focused 
behaviours. 

Additional  hypotheses  concern  the  relation  between  the  team-  and  task- focused  behaviours  and  the  team  and 
task  outcome  factors.  Research  has  shown  that  both  task-  and  team-focused  behaviours  contribute  to  mission 
outcomes  (see  review  in  Essens  et  al.,  2005).  More  specifically,  these  findings  suggest  a  relationship  between 
le\’els  of  the  process  variables  and  levels  of  the  outcome  variables.  Accordingly,  the  following  hypotheses 
will  be  examined: 

•  H2:  There  will  be  a  positive  relationship  between  task- focused  behaviours  and  task  outcomes; 

•  H3:  There  will  be  a  positive  relationship  between  team-focused  behaviours  and  team  outcomes; 

•  H4:  There  will  be  a  positive  relationship  between  task-focused  behaviours  and  team  outcomes;  and 

•  H5:  There  will  be  a  positive  relationship  between  team-focused  behaviours  and  task  outcomes. 

The  next  set  of  hypotheses  focuses  on  the  relative  importance  of  team-  vs.  task-focused  behaviours  with 
respect  to  outcomes.  Currently,  the  CTEF  model  does  not  view  team-  and  task-focused  behaviours  as 
independent  of  each  other.  A  similar  view  is  held  of  the  team  and  task  outcomes.  Thus,  failure  to  find 
support  for  this  assumption  would  necessitate  a  clearer  articulation  of  that  relation.  We  therefore 
hypothesized: 

•  H6:  Task-  and  team-focused  behaviours  will  be  positively  correlated  with  task  outcomes, 
but  task-focused  behaviours  will  be  more  highly  correlated;  and 

•  H7:  Task-  and  team-focused  behaviours  will  be  positively  correlated  with  team  outcomes,  but  team- 
focused  behaviours  will  be  more  highly  correlated. 

Hypothesis  8  provides  the  critical  assessment  of  the  central  assumption  of  a  mediational  role  of  team-  and 
task-focused  behaviours: 

•  H8:  The  relationships  between  conditions  and  task  and  team  outcomes  will  be  mediated  by  task-  and 
team-focused  behaviours  (i.e.,  conditions  processes  (task,  team)  outcomes  (task,  team)). 

The  final  set  of  hypotheses  that  bear  on  the  CTEF  model  will  examine  fundamental  assumptions  concerning 
the  role  of  feedback.  Indeed  the  CTEF  model  involves  several  key  feedback  loops  and  the  role  of  feedback  in 
task  and  team  effectiveness  is  explicit  (see  Essens  et  al.,  2005).  Accordingly,  we  hypothesized  that: 

•  H9:  Teams  that  used  feedback  processes  will  have  higher  scores  on  task  outcomes  than  teams  that 
did  not  use  feedback  processes; 

•  Hi0:  Teams  that  used  feedback  processes  will  have  higher  scores  on  team  outcomes  than  teams 
that  did  not  use  feedback  processes; 

•  Hu:  Teams  that  used  feedback  processes  will  have  higher  scores  on  task- focused  behaviours  than 
teams  that  did  not  use  feedback  processes;  and 

•  Hi2:  Teams  that  used  feedback  processes  will  have  higher  scores  on  team-focused  behaviours  than 
teams  that  did  not  use  feedback  processes. 

4.2.2  Instrument  Validation 

The  survey  study  focused  on  three  areas  of  instrument  validation: 

•  General  reactions  of  commanders  concerning  the  usefulness,  clarity,  and  coverage  of  the  instrument; 
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•  Correspondence  between  empirical  examination  of  relationships  between  aspects  of  the  CTEF 
model,  as  measured  with  the  CTEF  instrument,  and  relationships  accepted  in  the  broader 
psychological  research  literature;  and 

•  Psychometric  properties. 

With  regard  to  the  first  area,  several  questions  were  included  in  the  survey  collecting  respondent’s 
reactions  to  CTEF  instrument.  This  information  was  collected  using  a  mix  of  structured  and  unstructured 
response  items.  With  regard  to  the  second  area,  the  hypotheses  discussed  in  the  section  above  were 
examined.  In  terms  of  the  last  area  of  focus,  a  major  objective  was  to  assess  whether  we  can  reduce  the 
number  of  survey  items,  (i.e.,  is  there  redundancy  within  each  of  the  major  components  of  the  model?). 
The  issue  was  raised  by  the  end-user  communities  in  previous  applications  of  the  instrument  in  terms  of 
length  of  time  to  complete  and  the  sheer  number  of  items.  In  the  present  study  we  will  base  our  decisions 
concerning  item  reduction  on  traditional  methods  examining  correlation  patterns  across  items. 
For  example,  if  we  discover  that  certain  items  are  very  highly  correlated  with  other  items  then  they  can  be 
viewed  as  redundant  and  thus  can  be  eliminated  from  the  instrument.  In  addition,  we  examined  the  extent 
to  which  the  features  (Fevel  4)  within  the  team-  and  task-focused  behaviours  are  psychometrically  sound. 
To  the  extent  that  these  features  are  not  contributing  to  the  explanation  of  unique  variance  over  and  above 
that  explained  by  the  higher  level  item  (Fevel  3),  then  they  likewise  can  be  eliminated. 


4.3  METHOD 

4.3.1  Participants 

Participants  were  718  military  personnel  from  14  NATO  nations  with  operational  experience  as  leaders  or 
members  of  military  command  teams.  The  mean  age  of  the  sample  was  42.5  yrs  (Std.  Dev.  =  6.70;  range 
26  -  60  years).  Overall,  49/718  (6.82%)  of  the  participants  were  female.  Participant’s  officer  ranks  ranged 
from  Captain  to  Ft.  General  and  also  included  senior  non-commissioned  officers  who  served  as  members  of 
command  teams  in  operations.  Detailed  breakdowns  of  the  participants’  age,  nationality,  rank,  and  echelon 
are  provided  in  Tables  C-l  —  C-4  of  Annex  C. 

Most  participants  were  recruited  by  email,  although  direct  requests  for  participation  were  used  in  some 
military  educational  institutions.  The  response  rate  for  email  requests  was  558/7,848  (7.1%  response  rate). 
A  total  of  160  respondents  completed  the  pencil  and  paper  version  of  the  survey.  Participants  were  not 
compensated  for  completing  the  survey.  The  study  was  approved  by  the  Human  Research  Ethics  Boards  of 
Canada  (DRDC  Toronto),  US  Army  Research  Institute,  and  US  Naval  Air  Warfare  Center-Training  Systems 
Division.  The  study  also  was  reviewed  and  received  exempt  status  from  the  Belgian  Defence  Ethics  Board, 
the  TNO  Ethics  Committee  (Soesterberg,  Netherlands),  and  Finkoping  University  (which  represents  FOI  - 
Linkoping,  Sweden). 

4.3.2  Apparatus 

The  primary  form  of  data  collection  was  a  web-based  survey.  The  panel  representatives  of  Canada,  US, 
and  the  Netherlands  distributed  the  survey  to  military  members  of  their  respective  countries  by  wide  email 
distribution.  Only  participants  that  comprised  the  target  population  were  solicited  (i.e.,  operational 
experience  as  a  leader  or  member  of  a  command  team).  Participants  who  were  recruited  via  email 
distribution  were  invited  to  complete  a  web-based  version  of  the  instrument  housed  at  TNO  Defence, 
Security,  and  Safety  (Soesterberg,  NF)  or  the  U.S.  Army  Research  Institute  (Arlington,  VA,  USA)  (part  of 
this  web-based  version  of  the  CTEF  instrument  employed  in  the  present  study  is  provided  in  Annex  B). 
In  cases  where  email  data  collection  was  not  possible  (e.g.,  no  access  to  email  distribution  lists,  local 
organizational  constraints,  or  security  concerns),  members  of  the  NATO  panel  solicited  participation  at 
various  military  installations  (e.g.,  military  academies,  staff  colleges,  commands).  In  these  cases, 
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participants  completed  comparable  paper  copy  versions  of  the  instrument.  Obtaining  signed  informed 
consent  from  participants  who  completed  the  web-based  survey  was  not  feasible.  However,  the  web-based 
version  of  the  survey  did  provide  an  informed  consent  form  and  participants  were  required  to  indicate  that 
they  “Yes,  I  agree”  or  “No,  I  do  not  agree”  to  participate  in  the  study  by  clicking  a  button  next  to  their 
choice  at  the  bottom  of  the  electronic  consent  form.  Respondents  choosing  “No,  I  do  not  agree”  were 
thanked  and  excused  from  further  participation.  The  same  survey  was  used  for  all  participants.  The  web- 
based  version  of  the  survey  was  available  in  both  official  NATO  languages  (English  and  French). 
In  addition,  English,  French,  Dutch,  and  Bulgarian  hard-copy  versions  of  the  survey  were  also  prepared  in 
order  to  accommodate  non-web  administration  and  translation  requests  from  Belgian  and  Bulgarian 
participants,  respectively.  For  each  translation,  the  scale  items  that  comprise  the  CTEF  underwent 
extensive  discussion,  verification,  back  translation,  and  revisions  prior  to  administration. 

4.3.3  Procedure 

Participants  took  approximately  30  -  45  minutes  to  complete  the  survey.  First,  participants  read  an 
information  page  that  provided  a  brief  overview  of  the  aims  and  scope  of  the  study,  and  the  informed 
consent  form.  Participants  then  filled  out  a  variety  of  demographic  questions  (see  Annex  B).  They  were 
then  asked  to  reflect  on  a  command  team  they  were  part  of  in  a  recent  deployment  and  to  describe  the 
team  they  were  considering.  Participants  then  completed  the  CTEF  instrument  in  the  context  of  their 
experiences  as  a  member  or  leader  of  the  team  they  were  reflecting  upon.  The  survey  was  then  followed 
by  a  short  feedback  questionnaire  on  the  perceived  utility  of  the  instrument.  Results  of  this  feedback  are 
also  discussed. 

The  overarching  construct  being  assessed  with  the  survey  is  command  team  effectiveness.  To  capture  this, 
team  effectiveness  was  broken  down  into  the  specific  concepts  in  the  CTEF  model.  Concepts  being 
assessed  via  subjective  ratings  were  mission  framework,  task  characteristics,  organisational  characteristics, 
team  leader  characteristics  and  capabilities,  team  member  characteristics  and  capabilities,  overall  team 
characteristics,  task-focused  behaviours,  team-focused  behaviours,  task  outcomes,  team  outcomes, 
and  feedback  behaviours.  Each  of  these  concepts  was  measured  with  between  3  and  19  items.  Participants 
were  asked  to  respond  to  each  item  using  the  two  rating  scales  magnitude  and  impact.  Both  of  these  rating 
scales  were  used  for  all  items  representing  the  CTEF  model,  with  the  exception  of  the  items  task  and  team 
outcomes,  which  had  only  the  magnitude  scale.  The  web-based  version  of  the  survey  showed  the  answer 
categories  in  a  pull-down  menu  list,  the  text  version  used  text  boxes  that  could  be  marked. 


4.4  RESULTS 

Data  were  aggregated  into  a  common  format  across  national  surveys.  Item  analyses,  content  analyses, 
and  all  verification  analyses  were  conducted  using  professionally  accepted  statistical  software  (e.g.,  SPSS, 
AMOS).  For  the  most  part,  analyses  involved  correlation,  regression,  Analyses  of  Variance  (ANOVA), 
and  Structural  Equation  Modeling  (SEM). 

4.4.1  Command  Team  Demographics:  The  Operational  Environment 

As  mentioned  previously,  Tables  C-l  -  C-4  of  Annex  C  provide  detailed  breakdowns  of  demographic  data 
for  age,  rank,  echelon,  and  nationality  of  the  participants  in  the  study.  Overall,  the  data  cover  a  very  broad 
spectrum  of  backgrounds  and  thus  provide  effective  representation  of  command  team  membership. 
In  addition  to  the  demographic  data  in  Tables  C-l  -  C-4,  Annex  C  also  provides  data  on  two  demographic 
questions  that  provided  information  on  the  various  services  and  nationalities  of  the  command  team 
members  with  whom  they  worked  in  operations.  Tables  C-5  and  C-6  of  Annex  C  provide  the  results  of 
these  questions.  Table  C-5  provides  a  broader  perspective  of  the  joint,  and  in  some  cases  interagency, 
collaborative  context  within  which  modem  command  teams  must  operate.  Table  C-6  is  particularly 
noteworthy  in  showing  that  participants  in  the  current  study  participated  with  over  2,500  command  team 
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leaders  and  members  from  over  85  Nations,  highlighting  the  truly  multi-national  nature  of  contemporary 
operations. 

Another  demographic  question  probed  the  degree  of  interdependence  among  the  command  team  members 
with  which  they  participated  on  a  5-point  scale:  i.e.,  “Not  at  all”,  “a  little”,  “somewhat”,  “a  lot”,  and  “very 
much”.  A  breakdown  of  the  data  is  provided  in  Table  4-2.  As  is  evident,  approximately  90%  of  the 
participants  felt  their  teams  were  at  least  “somewhat”  interdependent;  moreover,  approximately  65%  felt 
that  their  teams  were  highly  interdependent  (i.e.,  “a  lot”  or  “very  much”).  These  results  serve  to  illustrate 
and  foreshadow  the  potential  significance  of  the  team  process  factors  in  command  team  effectiveness. 

Table  4-2:  Distribution  of  Command  Team  Interdependence. 


Frequency 

Percent 

Not  at  all 

9 

1.25 

A  little 

50 

6.96 

Somewhat 

187 

26.04 

A  lot 

301 

41.92 

Very  much 

168 

23.40 

Total 

715 

99.58 

Other 

3 

0.42 

A  final  demographic  question  concerned  the  distributional  nature  of  the  environment  within  which  the 
command  team  typically  worked  (i.e.,  exclusively  or  largely  face-to-face  teams  versus  exclusively  or  largely 
distributed  teams).  Overall,  471  (65.6%)  of  participants  reported  that  their  command  teams  were  co-located, 
222  (30.92%)  were  partially  distributed,  and  only  25  (3.48%)  reported  working  in  a  fully  distributed  team 
context.  Evidently,  operational  command  teams  work  in  a  highly  face-to-face  collaborative  context. 

4.4.2  Instrument  Item  Analyses 
4.4.2. 1  Level  versus  Impact  Analyses 

The  first  analysis  concerns  the  relationship  between  judgments  of  level  or  magnitude  and  judgments  of 
impact.  As  discussed  previously,  the  2005  version  of  CTEF  made  a  clear  distinction  between  magnitude  and 
impact  assessments:  magnitude  was  to  denote  the  extent  to  which  a  constituting  aspect  was  evident  in  their 
command  team  (e.g.,  operational  stress  level),  and  impact  was  to  denote  the  extent  to  which  the  aspect  at  that 
magnitude  level  influenced  the  effectiveness  of  the  command  team.  In  practice,  however,  we  have  found 
very  high  correlations  between  magnitude  and  impact  assessments  (e.g.,  Joint  Caribbean  Lion  >  0.75). 
Hence,  we  are  faced  with  the  reality  that  commanders  and  team  members  might  find  the  distinction  useful 
but  unclear  (see  the  feedback  from  respondents  below),  or  else  the  two  assessments  simply  cannot  be  easily 
distinguished.  As  in  the  previous  studies,  the  overall  correlation  between  all  magnitude  and  impact 
assessments  was  very  high  in  the  present  study  ( r  =  0.7 1 9,  p  <  .0001).  In  addition,  as  is  evident  in  Table  4-3, 
this  relationship  held  when  the  data  were  broken  down  by  team-focused  behaviours  (r  =  0.818,  p  <  .0001), 
task-focused  behaviours  (r  =  0.8 1 8,  p  <  .0001),  team  condition  components  (i.e.,  leader,  member,  team; 
r  =  0.773,  p  <  .0001),  and  task  condition  components  (i.e.,  mission  framework,  task,  and  organization; 
r  =  0.379,  p<. 0001). 
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Table  4-3:  Distribution  of  Magnitude  and  Impact  Responses  Across  Task  Conditions 
(Mission  Framework,  Task,  and  Organization),  Task-Focused  Behaviours, 

Team  Conditions  (Leader,  Member,  Team),  and  Team-Focused  Behaviours. 

Impact 

•  2 
Magnitude 

Very 

Negative 

Negative 

Neutral 

Positive  _V®J7 
Positive 

Total 

Task  Conditions  N/A 

1 

0 

7 

3 

0 

11 

Very  Low 

237 

232 

75 

198 

60 

802 

Low 

140 

758 

254 

509 

24 

1685 

Moderate 

41 

922 

994 

728 

24 

2709 

High 

28 

208 

726 

1019 

133 

2114 

Very  High 

20 

48 

364 

224 

309 

965 

Total 

467 

2168 

2420 

2681 

550 

8286 

Task-Focused  Behaviours  N/A 

0 

0 

3 

2 

0 

5 

Very  Low 

263 

59 

4 

0 

0 

326 

Low 

89 

829 

74 

11 

2 

1005 

Moderate 

4 

748 

1541 

1300 

24 

3617 

High 

2 

44 

362 

5538 

293 

6239 

Very  High 

0 

3 

42 

749 

1478 

2272 

Total 

358 

1683 

2026 

7600 

1797 

13464 

Team  Conditions  N/A 

0 

0 

6 

1 

0 

7 

Very  Low 

191 

82 

20 

8 

1 

302 

Low 

75 

492 

125 

59 

3 

754 

Moderate 

7 

420 

776 

886 

22 

2111 

High 

8 

55 

218 

3263 

424 

3968 

Very  High 

11 

15 

15 

330 

1008 

1379 

Total 

292 

1064 

1160 

4547 

1458 

8521 

Team-Focused  Behaviours  N/A 

1 

0 

8 

4 

0 

13 

Very  Low 

285 

122 

14 

3 

0 

424 

Low 

75 

921 

166 

25 

5 

1192 

Moderate 

15 

574 

1751 

1382 

33 

3755 

High 

1 

40 

310 

4694 

413 

5458 

Very  High 

0 

10 

31 

496 

1398 

1935 

Total 

377 

1667 

2280 

6604 

1849 

12777 

2 

Some  of  the  scale  anchors  had  different  labels.  For  ease  of  use  we  used  the  Very  low  -  Very  high  labels  in 
representing  the  1-5  Likert  scale. 

the  data  presentation 
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Based  on  the  results  of  this  preliminary  analysis,  all  subsequent  analyses  will  be  conducted  on  magnitude 
assessments  only.  Again,  we  chose  magnitude  assessments  over  impact  assessments  because  all  of  the 
outcome  assessments  are  made  on  magnitude  judgments  only,  thus  providing  consistency  of  comparison 
throughout. 

4.4.2.2  Item  and  Scale  Analyses 

Each  of  the  major  concepts  within  the  CTEF  model  was  assessed  by  a  number  of  items  (3  to  19  items). 
Although  in  the  creation  of  these  items  they  were  designed  to  tap  into  different  and  unique  aspects  of  these 
concepts,  they  are  also  believed  to  be  summative  indicators  of  the  overarching  concept.  As  such,  we  would 
expect  there  to  be  moderate  inter-item  correlations  and  moderate  reliability  estimates  as  computed  using 
the  Cronbach  a  statistic.  Table  4-4  reports  item  descriptive  statistics,  inter-item  correlations,  and  Cronbach 
a  for  each  concept. 

Within  the  task  condition  components  (i.e.,  mission  framework,  task,  and  organization),  the  descriptive  and 
inter-item  statistics  generally  supported  the  approach  of  summative  measurement.  Mission  framework  items 
exhibited  an  acceptable  reliability  (a  =  .59)  and  relatively  low  inter-item  correlations.  Task  characteristic 
items  also  exhibited  an  acceptable  reliability  (a  =  .55),  however  the  inter-item  correlations  indicated  two 
clusters  of  items.  Task  complexity  and  workload  correlated  highly  with  each  other  (r  =  .42),  as  did  goal 
ambiguity  and  goal  instability  (r  =  .61),  however  these  two  pairs  of  items  did  not  correlate  with  each  other 
very  strongly  at  all.  Organization  items  also  exhibited  an  acceptable  reliability  (a  =  .62)  with  fairly  high 
inter-item  correlations.  Notably,  the  autonomy  item  appeared  to  correlate  less  strongly  with  the  other  items. 

Within  the  team  condition  components  (i.e.,  team  leader  characteristics,  team  member  characteristics, 
and  team  characteristics),  the  descriptive  and  inter-item  statistics  generally  indicated  that  the  team  leader 
and  member  factors  were  assessed  with  parallel  items  rather  than  summative  items.  For  these  factors, 
the  reliability  estimates  were  high  (a  =  .87  and  a  =  .81  respectively),  and  the  inter-item  correlations  were 
correspondingly  high.  For  the  third  factor,  team  characteristics,  the  descriptive  and  inter-item  statistics 
generally  indicate  summative  measurement  with  an  acceptable  but  relatively  high  reliability  (a  =  .68)  and 
moderate  inter-item  correlations  with  the  exception  of  the  team  size  item.  This  item  did  not  correlate  with 
team  distribution,  match  of  team  goals  to  organizational  goals,  or  team  maturity. 

Within  the  task-  and  team-focused  behaviour  components,  the  descriptive  and  inter-item  statistics  strongly 
indicated  that  these  factors  were  assessed  with  parallel  items  rather  than  summative  items.  Inter-item 
correlations  were  quite  high  within  each  of  the  task-  and  team-focused  behaviour  components,  with  slightly 
higher  inter-item  correlations  within  specific  aspects  of  each  factor  (see  Table  4-4,  task-focused  behaviour 
sub-table,  team-focused  behaviour  sub-table).  Within  each  component,  several  aspects  were  assessed  directly 
and  with  several  items  targeting  sub-elements  (features)  of  each  aspect.  For  example,  within  task-focused 
behaviour,  the  aspect  ‘managing  information’  was  assessed  directly  (Item  1)  and  with  several  items  targeting 
features  of  ‘managing  information’  (Items  2,  3  and  4).  Feature  items  typically  correlated  slightly  higher  with 
each  other  than  with  items  assessing  other  aspects.  This  pattern  of  inter-item  correlations  was  consistent 
across  both  the  task-  and  team-focused  behaviour  components.  However,  even  across  aspects,  the  inter-item 
correlations  were  quite  high,  indicating  the  possibility  that  these  aspects  are  highly  related  to  one  another. 
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Table  4-4:  Descriptive  Statistics,  Inter-Item  Correlations,  and  Reliability  Statistics 
of  the  Conditions,  Outcomes,  Feedback,  and  Processes  Components. 

PROCESSES  -  Task-Focused  Behaviours 


a  =  0.953;  N  =  642 

Mean 

SD 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

1  -  Managing  Information 

3.70 

0.95 

- 

2  -  Obtaining  Information 

3.61 

0.96 

0.63 

- 

3  -  Processing  Information 

3.71 

0.85 

0.64 

0.60 

- 

4  -  Exchanging  Information 

3.57 

0.99 

0.66 

0.51 

0.58 

- 

5  -  Situational  Understanding 

3.65 

0.96 

0.56 

0.53 

0.62 

0.61 

- 

6  -  Decision  Making 

3.76 

0.88 

0.59 

0.49 

0.60 

0.58 

0.64 

- 

7  -  Detennining  the  Problem 

3.58 

0.91 

0.56 

0.49 

0.55 

0.57 

0.62 

0.71 

- 

8  -  Making  Timely  Decisions 

3.81 

0.91 

0.51 

0.49 

0.52 

0.53 

0.53 

0.63 

0.59 

- 

9  -  Evaluating  Options/Results 

3.65 

0.96 

0.53 

0.42 

0.54 

0.53 

0.58 

0.61 

0.60 

0.66 

- 

10  -  Planning 

3.75 

0.87 

0.55 

0.47 

0.50 

0.54 

0.61 

0.61 

0.59 

0.54 

0.58 

- 

1 1  -  Identifying  Future  Tasks 

3.52 

0.91 

0.46 

0.43 

0.44 

0.47 

0.56 

0.54 

0.52 

0.47 

0.54 

0.66 

- 

12  -  Scheduling 

3.57 

0.92 

0.46 

0.42 

0.46 

0.50 

0.51 

0.49 

0.49 

0.48 

0.50 

0.62 

0.61 

- 

1 3  -  Detennining  Resources 

3.68 

0.91 

0.44 

0.41 

0.44 

0.48 

0.49 

0.50 

0.46 

0.45 

0.49 

0.55 

0.54 

0.56 

- 

14  -  Mission  Approach 

3.58 

0.93 

0.47 

0.42 

0.48 

0.50 

0.55 

0.56 

0.51 

0.51 

0.55 

0.56 

0.54 

0.59 

0.56 

- 

15  -  Plan  Execution 

3.85 

0.81 

0.51 

0.44 

0.52 

0.48 

0.53 

0.53 

0.46 

0.49 

0.47 

0.53 

0.43 

0.43 

0.46 

0.48 

- 

1 6  -  Organising 

3.72 

0.86 

0.53 

0.42 

0.54 

0.54 

0.54 

0.58 

0.55 

0.52 

0.56 

0.58 

0.49 

0.54 

0.51 

0.53 

0.69 

- 

1 7  -  Managing 

3.69 

0.87 

0.50 

0.42 

0.52 

0.49 

0.52 

0.58 

0.56 

0.49 

0.55 

0.56 

0.47 

0.52 

0.51 

0.55 

0.58 

0.70 

- 

18  -  Monitoring  Progress 

3.61 

0.96 

0.48 

0.40 

0.50 

0.48 

0.53 

0.55 

0.53 

0.47 

0.54 

0.54 

0.48 

0.51 

0.47 

0.53 

0.58 

0.62 

0.62 

- 

19  -  Interactions  with  Other  Command  Teams 

3.70 

1.05 

0.44 

0.40 

0.40 

0.40 

0.43 

0.41 

0.40 

0.39 

0.38 

0.44 

0.41 

0.36 

0.31 

0.38 

0.42 

0.42 

0.39 

0.41 

Note:  Highlighted  rows  indicate  higher  level  items.  Sub-elements  of  items  are  indented. 
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PROCESSES  -  Team-Focused  Behaviours 


a  =  0.945;  N  =  657 

Mean 

SD 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

17 

1  -  Providing/Maintaining  Vision 

3.59 

0.91 

2  -  Maintaining  Common  Intent 

3.65 

0.92 

0.69 

3  -  Collaboration 

3.73 

0.95 

0.49 

0.52 

4  -  Communication 

3.74 

0.97 

0.45 

0.47 

0.69 

5  -  Coordination 

3.66 

0.89 

0.46 

0.50 

0.61 

0.62 

6  -  Providing  Feedback 

3.40 

1.03 

0.46 

0.50 

0.57 

0.55 

0.61 

7  -  Motivating 

3.55 

0.99 

0.51 

0.54 

0.49 

0.47 

0.49 

0.52 

8  -  Rewarding  and  Recognition 

3.41 

1.06 

0.47 

0.46 

0.48 

0.48 

0.44 

0.50 

0.62 

9  -  Pride  in  the  Work 

3.75 

1.01 

0.48 

0.56 

0.50 

0.47 

0.45 

0.46 

0.68 

0.59 

10  -  Adapting  to  Changes 

3.70 

0.87 

0.43 

0.47 

0.42 

0.39 

0.40 

0.41 

0.45 

0.40 

0.46 

1 1  -  Monitoring  Performance 

3.48 

0.82 

0.42 

0.46 

0.44 

0.39 

0.43 

0.47 

0.45 

0.49 

0.49 

0.50 

12  -  Correcting  Team  Members 

3.44 

0.91 

0.36 

0.38 

0.40 

0.38 

0.39 

0.43 

0.40 

0.41 

0.42 

0.43 

0.52 

1 3  -  Providing  Back-up 

3.69 

1.02 

0.39 

0.46 

0.45 

0.41 

0.47 

0.43 

0.48 

0.44 

0.53 

0.49 

0.44 

0.50 

14  -  Maintaining  Team  Synergy 

3.54 

0.92 

0.54 

0.57 

0.54 

0.54 

0.54 

0.47 

0.55 

0.51 

0.54 

0.47 

0.45 

0.41 

0.54 

15  -  Providing  Social  Support 

3.65 

0.98 

0.40 

0.44 

0.44 

0.46 

0.44 

0.42 

0.47 

0.52 

0.45 

0.41 

0.41 

0.32 

0.44 

0.65 

1 6  -  Regulating  Emotions 

3.38 

0.88 

0.42 

0.43 

0.40 

0.43 

0.46 

0.43 

0.43 

0.46 

0.43 

0.42 

0.38 

0.35 

0.43 

0.57 

0.61 

17  -  Maintaining  Cohesion 

3.65 

1.00 

0.51 

0.55 

0.51 

0.54 

0.52 

0.48 

0.55 

0.53 

0.56 

0.48 

0.44 

0.44 

0.49 

0.69 

0.63 

0.62 

18  -  Handling  Conflicts 

3.33 

0.99 

0.50 

0.49 

0.49 

0.52 

0.51 

0.50 

0.51 

0.53 

0.50 

0.44 

0.45 

0.42 

0.47 

0.64 

0.54 

0.62 

0.62 

Note:  Highlighted  rows  indicate  higher  level  items.  Sub-elements  of  items  are  indented. 
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CONDITIONS  -  Mission  Framework 


a  =  0.594;  N  =  653 

Mean 

SD 

(1) 

(2) 

(3) 

(4) 

(1)  Situational  Uncertainty 

2.69 

1.08 

- 

0.228 

0.290 

0.132 

(2)  Operational  Stress 

3.14 

1.05 

0.228 

- 

0.336 

0.353 

(3)  Constraints 

3.25 

1.06 

0.290 

0.336 

- 

0.265 

(4)  Mission  Stakes 

3.66 

1.02 

0.132 

0.353 

0.265 

- 

CONDITIONS  -  Task 


a  =  0.553;  N  =  616 

Mean 

SD 

(1) 

(2) 

(3) 

(4) 

(1)  Task  Complexity 

3.38 

0.90 

- 

0.417 

0.106 

0.138 

(2)  Workload 

3.56 

0.98 

0.417 

- 

-0.003 

0.092 

(3)  Goal  Ambiguity 

2.65 

1.20 

0.106 

-0.003 

- 

0.606 

(4)  Goal  Instability 

2.67 

1.17 

0.138 

0.092 

0.606 

- 

CONDITIONS  -  Organisation 


a  =  0.618;  N  =  684 

Mean 

SD 

(1) 

(2) 

(3) 

(4) 

(1)  Team  Mission  Match 

3.74 

0.98 

- 

0.429 

0.216 

0.309 

(2)  Clarity  of  Command 

3.54 

1.13 

0.429 

- 

0.167 

0.357 

(3)  Freedom  of  Action 

3.41 

1.07 

0.216 

0.167 

- 

0.285 

(4)  Organisation  Support 

3.35 

1.03 

0.309 

0.357 

0.285 

- 

CONDITIONS  -  Team  Leader 


a  =  0.868;  N  =  697 

Mean 

SD 

(1) 

(2) 

(3) 

(1)  Skills 

3.76 

1.08 

- 

0.752 

0.679 

(2)  Knowledge 

3.79 

1.03 

0.752 

- 

0.639 

(3)  Org  Match 

3.68 

1.07 

0.679 

0.639 

- 

CONDITIONS 

-  Team  Member 

a  =  0.814;  N  =  695 

Mean 

SD 

(1) 

(2) 

(3) 

(1)  Skills 

3.75 

0.81 

- 

0.730 

0.531 

(2)  Knowledge 

3.70 

0.82 

0.730 

- 

0.532 

(3)  Org  Match 

3.47 

0.87 

0.531 

0.532 

- 
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CONDITIONS  -  Team 


a  =  0.676;  N  =  692 

Mean 

SD 

(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(l)Mix  of  People 

3.82 

0.86 

- 

0.200 

0.343 

0.412 

0.197 

0.305 

(2)  Size 

3.19 

0.98 

0.200 

- 

0.348 

0.118 

0.056 

0.075 

(3)  Structure 

3.46 

1.02 

0.343 

0.348 

- 

0.301 

0.237 

0.325 

(4)  Maturity 

3.38 

1.06 

0.412 

0.118 

0.301 

- 

0.275 

0.333 

(5)  Face  to  Face 

3.86 

0.91 

0.197 

0.056 

0.237 

0.275 

- 

0.350 

(6)  Org  Match 

3.73 

0.86 

0.305 

0.075 

0.325 

0.333 

0.350 

- 

OUTCOMES  -  Task 


a  =  0.  813;  N  =  694 

Mean 

SD 

(1) 

(2) 

(3) 

(1)  Cmdr/Org  Goals 

4.07 

0.84 

- 

0.630 

0.598 

(2)  Partner  Expectations 

3.96 

0.85 

0.630 

- 

0.538 

(3)  Mission  Limits 

4.17 

0.84 

0.598 

0.538 

- 

OUTCOMES  -  Team 


a  =  0.  927;  N  =  712 

Mean 

SD 

1 

2 

3 

4 

5 

6 

(1)  Trust 

3.79 

0.96 

- 

0.721 

0.759 

0.631 

0.606 

0.693 

(2)  Morale 

3.84 

1.00 

0.721 

- 

0.748 

0.666 

0.631 

0.654 

(3)  Cohesion 

3.73 

1.04 

0.759 

0.748 

- 

0.692 

0.666 

0.728 

(4)  Confidence 

3.89 

0.97 

0.631 

0.666 

0.692 

- 

0.719 

0.610 

(5)  Shared  Vision 

3.67 

0.97 

0.606 

0.631 

0.666 

0.719 

- 

0.615 

(6)  Respect 

3.82 

0.98 

0.693 

0.654 

0.728 

0.610 

0.615 

- 

FEEDBACK  -  Task  and  Team 


a  =  0.  954;  N  =  694 

Mean 

SD 

1 

2 

3 

4 

5 

6 

(1)  Task  Goal  Achievement 

4.32 

1.20 

- 

0.781 

0.735 

0.837 

0.736 

0.731 

(2)  Improve  Task  Processes 

4.44 

1.17 

0.781 

- 

0.802 

0.758 

0.796 

0.721 

(3)  Improve  Task  Conditions 

4.38 

1.22 

0.735 

0.802 

- 

0.761 

0.765 

0.775 

(4)  Team  Goal  Achievement 

4.29 

1.20 

0.837 

0.758 

0.761 

- 

0.793 

0.791 

(5)  Improve  Team  Processes 

4.34 

1.17 

0.736 

0.796 

0.765 

0.793 

- 

0.805 

(6)  Improve  Team  Conditions 

4.27 

1.21 

0.731 

0.721 

0.775 

0.791 

0.805 

- 
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Within  the  task  and  team  outcome  factors,  the  descriptive  and  inter-item  statistics  strongly  indicated  that 
these  factors  were  assessed  with  parallel  items  rather  than  summative  items.  Similar  to  the  team-  and  task- 
focused  behaviours,  the  inter-item  correlations  were  quite  high  within  each  factor.  Both  the  task  and  team 
outcome  factors  had  high  reliability  estimates  (a  =  .81  and  a  =  .93  respectively).  Although  the  items  for 
each  of  these  factors  were  designed  to  assess  unique  aspects  of  these  factors,  it  appears  that  in  practice  the 
items  are  largely  assessing  unitary  or  highly  interrelated  concepts. 

The  feedback  factor  was  originally  intended  to  directly  assess  several  feedback  behaviours,  distinguishing 
between  reviewing  performance  and  taking  steps  to  improve  behaviour  in  specific  areas  impacting  team 
effectiveness.  If  these  items  functioned  as  intended,  we  should  observe  a  moderate  to  low  reliability 
estimate  when  considering  these  items  as  a  single  scale  and  low  inter-item  correlations.  Converse  to  our 
expectations,  these  items  exhibited  very  high  inter-item  correlations  and  very  high  reliability  (a  =  .95). 
As  a  result,  these  items  were  treated  as  a  single  scale  for  analysis  purposes  in  this  study.  However,  as  will 
be  discussed  later,  we  also  recommend  further  review  of  the  assessment  of  feedback  processes  in  military 
populations  as  we  believe  these  items  were  demonstrated  to  be  inadequate  for  full  and  accurate  assessment 
of  these  processes. 

4.4.3  Tests  of  Hypotheses 

This  section  reviews  the  analytical  tests  of  the  12  a  priori  hypotheses  designed  to  assess  the  fundamental 
predictions  of  the  CTEF  model  (see  the  overview  in  Table  4-1  above).  Hypothesis  1  predicted  that  the 
condition  factors  will  be  correlated  with  the  task-  and  team-focused  behaviours.  This  hypothesis  was 
supported,  with  the  exception  of  mission  framework,  which  was  not  significantly  correlated  with  the  task- 
and  team-focused  behaviours  (see  Table  4-5). 

Table  4-5:  Correlations  between  Condition  Factors  and  Task-  and  Team-Focused  Behaviours. 


Task-Focused 

Behaviours 

Team-Focused 

Behaviours 

Mission  Framework 

-0.021 

-0.021 

Task 

-0.136** 

-0.086* 

Organization 

0.514** 

0.478* 

Team  Feader 

0.578** 

0.567* 

Team  Member 

0.505** 

0.498* 

Team 

0.621** 

0.597* 

*p  <  .05,  **p  <  .01,  and  ***p  <  .001 


Hypothesis  2  predicted  that  task-focused  behaviours  would  be  correlated  with  the  task  outcomes. 

Hypothesis  3  predicted  that  team-focused  behaviours  would  be  correlated  with  the  team  outcomes. 

Hypothesis  4  predicted  that  task-focused  behaviours  would  be  correlated  with  the  team  outcomes. 

Hypothesis  5  predicted  that  team-focused  behaviours  would  be  correlated  with  the  task  outcomes.  These 

hypotheses  were  all  supported  (see  Table  4-6). 


Table  4-6:  Correlations  between  Processes  and  Outcomes. 


Task  Outcomes 

Team  Outcomes 

Task-Focused  Behaviours 

.706** 

.748** 

Team-Focused  Behaviours 

.653** 

.867** 

*p  <  .05,  **p  <  .01,  ***p  <  .001 
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Hypothesis  6  predicted  that  task-  and  team-focused  behaviours  would  be  positively  correlated  with  task 
outcomes,  but  task-focused  behaviours  would  be  more  highly  correlated.  This  prediction  was  supported. 
The  correlation  between  task  focussed  behaviours  and  task  outcomes  (r  =  .706)  was  significantly  higher 
(Z  =  1.86,  p  <  .05)  than  the  correlation  between  team-focused  behaviours  and  task  outcomes  (r  =  .653). 
Similarly,  Hypothesis  7  predicted  that  task-  and  team-focused  behaviours  would  be  positively  correlated 
with  team  outcomes,  but  team-focused  behaviours  would  be  more  highly  correlated.  This  prediction  was 
likewise  supported.  The  correlation  between  team-focused  behaviours  and  team  outcomes  (r  =  .867)  was 
significantly  higher  (Z  =  6.66,  p  <  .001)  than  the  correlation  between  task-focused  behaviours  and  team 
outcomes  (r  =  .748). 

Formally  stated,  Hypothesis  8  predicts  a  mediating  effect  of  the  task-  and  team-focused  behaviours  on  the 
relationship  between  the  conditions  and  the  outcomes  (cf.  Mathieu,  Heffner,  Goodwin,  Salas  and  Cannon- 
Bowers,  2000;  Ilgen,  Hollenbeck,  Johnson,  and  Junt,  2005;  Mathieu  and  Schultze,  2006).  This  prediction 
was  tested  using  the  four-step  regression  technique  specified  by  Baron  and  Kenny  (1986).  In  the  first  step, 
the  relationship  is  assessed  between  the  predictor  and  criterion  (i.e.,  X  Y).  In  the  second  step  the 
relationship  is  assessed  between  the  predictor  and  the  mediator  (X  Z).  In  the  third  step,  the  relationship  is 
assessed  between  the  mediator  and  the  criterion  (i.e.,  Z  Y).  Those  relationships  are  expected  to  be 
statistically  significant.  In  the  final  step,  the  relationship  between  the  predictor  and  the  criterion  should 
become  zero  in  the  presence  of  the  mediator  (X  Z  Y).  From  a  practical  standpoint,  this  process  is 
executed  with  a  series  of  four  multiple  regression  analyses,  with  the  final  step  utilizing  a  multi-step  analysis 
entering  the  mediating  variables  in  the  first  step  and  the  predictor  variables  in  the  second  step. 

Each  regression  step  is  performed  twice;  i.e.,  once  with  respect  to  task  variables  and  once  with  respect  to 
team  variables.  Throughout  this  section,  an  *  following  a  regression  weight  indicates  that  this  value  is 
significant  at  the  p  <  0.05  level.  In  the  figures,  significant  regression  weights  are  plotted  in  bold  lines  and 
non  significant  regression  weights  are  plotted  by  a  dotted  line.  Unless  reported  otherwise,  the  regression 
weights  are  the  standardized  regression  weights  {ft). 


Step  1:  X  (Conditions)  — >  Y  (Outcomes) 

Using  all  six  condition  aspects  (i.e.,  mission  framework,  task  characteristics,  organisation  characteristics, 
team  leader,  team  member,  and  team  characteristics)  as  predictors  for  the  regression  with  task  outcomes  as 
criteria  results  in  a  PVAF3  =  37%  [R2  =  .374,  F{6,  710)  =  70.813,  p  <  0.001].  This  means  that  63%  is 
measurement  error  or  is  to  be  explained  by  other  factors  that  yet  do  not  appear  as  predictors  in  the  model. 

The  regression  weights,  shown  in  Figure  4- 1 ,  are  all  significant  except  for.  This  indicates  that  the  conditions 
that  have  an  impact  on  task  outcomes  are,  in  order  of  size:  team  leader,  team  characteristics,  organisation 
characteristics,  team  member,  and  task  characteristics.  It  is  important  to  note  that  all  significant  predictors 
but  one  has  a  positive  impact  on  the  task  outcomes;  i.e.,  the  negative  value  for  task  characteristics  indicates 
that  higher  values  on  task  characteristics  result  in  lower  task  outcomes. 


3 

PVAF:  Proportion  Variance  Accounted  For.  PVAF  is  a  measure  of  how  much  of  the  total  variance  of  the  dependent  variable 
(the  criterion)  is  explained  for  by  the  independent  variables  (the  predictors). 
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Using  all  six  condition  aspects  as  predictors  for  the  regression  on  team  outcomes  as  criteria  leads  to  a 
PVAF  =  53%  [R2  =  .527,  F( 6,  710)  =  131.728,  p  <  0.001].  In  other  words,  47%  of  the  variance  is  due  to 
measurement  error  or  variables  that  are  not  yet  included  in  the  model  as  predictors.  The  regression 
weights,  shown  in  Figure  4-2,  are  all  significant.  The  conditions  that  have  a  positive  impact  on  team 
outcomes  are,  in  order  of  size:  team  characteristics,  team  leader,  team  member,  organisation 
characteristics.  Task  characteristics  and  mission  framework  have  a  negative  impact  on  the  outcomes. 


Step  2:  X  (Conditions)  — »  Z  (Processes) 

Using  all  six  condition  aspects  as  predictors  for  the  regression  on  task-focused  behaviours  as  criteria  results 
in  a  PVAF  =  55%  \R2=  .551,  F( 6,  710)  =  145.008,  p  <  0.000].  In  other  words,  45%  is  due  to  measurement 
error  or  variables  that  are  not  yet  included  in  the  model  as  predictors.  The  regression  weights  (see  Figure  4-3) 
are  all  significant  except  for  mission  framework.  The  condition  aspects  that  have  a  positive  impact  on  task- 
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focused  behaviours  are,  in  order  of  size:  team  leader,  team  characteristics,  team  member,  organisation 
characteristics.  Task  characteristics  have  a  negative  effect  on  the  task- focused  behaviours. 


Figure  4-3:  Regression  Model  for  Task-Focused  Behaviours  using  Conditions  as  Predictors. 

Using  all  six  conditions  as  predictors  for  the  regression  on  team-focused  behaviours  as  criterion  leads  to  a 
PVAF  =  51%  [R2  =  .510,  F( 6,  710)  =  123.227,  p  <  0.000].  The  regression  weights  (see  Figure  4-4)  are  all 
significant  except  for  mission  framework  and  task  characteristics.  The  condition  aspects  that  have  a 
positive  impact  on  team  processes  are,  in  order  of  size:  team  leader,  team  characteristics,  team  member, 
and  organisation  characteristics.  No  predictors  have  a  negative  impact. 


Figure  4-4:  Regression  Model  for  Team-Focused  Behaviours  using  Conditions  as  Predictors. 


Step  3:  Z  (Processes)  — >  Y  (Outcomes) 

Using  both  the  task-  and  team-focused  behaviours  as  predictors  for  the  regression  on  task  outcomes 
as  criteria  gives  a  PVAF  =  52%  [7?2  =  .520,  F( 2,  714)  =  387.503,  p  <  0.000].  The  regression  weights 
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(see  Figure  4-5)  are  both  significant  but  are  notably  different;  the  impact  of  task- focused  behaviours  on  the 
task  outcomes  is  twice  as  large  (.505)  as  the  impact  of  the  team-focused  behaviours  (.252). 


Figure  4-5:  Regression  Model  for  Task  Outcomes  using  the  Processes  as  Predictors. 

Using  both  processes  as  predictors  for  the  regression  on  team  outcomes  results  in  a  PVAF  =  76% 
\R2  =  .761,  F( 2,  714)  =  1 135.349,  p  <  0.000],  The  regression  weights  (see  Figure  4-6)  are  both  significant. 
Again,  it  should  be  noted  that  the  effect  of  team-focused  behaviours  on  team  outcomes  is  considerably 
larger  than  the  effect  of  task-focused  behaviours  (.739  vs.  .161). 


Figure  4-6:  Regression  Model  for  Team  Outcomes  using  the  Processes  as  Predictors. 

Step  4:  X  (Conditions)  — *  Z  (Processes)  — >  Y  (Outcomes) 

If  the  task-  and  team-focused  behaviours  fully  mediate  the  relationship  between  conditions  and  task 
outcomes,  then  the  regression  weights  of  conditions  on  the  task  outcomes,  in  the  presence  of  the 
mediational  variables  task-  and  team-focused  behaviours,  should  be  zero  or  approach  zero.  For  this 
analysis  the  PVAF  =  54%  [R2  =  .535,  F( 8,  708)  =  101.980,  p  <  0.000],  an  increase  of  approximately  2% 
PVAF  (AR2  =  .015,  A(6,708)  =  3.784,  p  <  .01).  All  regression  weights  originating  in  the  conditions  are 
now  non-significant  except  for  team  leader  (/?  =  0.120*)  which  was  expected  to  be  non-significant.  In  our 
model,  we  observe  indeed  a  mediating  effect  of  the  processes,  but  there  is  still  some  direct  effect  present 
of  team  leader  characteristics  (see  Figure  4-7). 
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Figure  4-7:  Full  Mediating  Regression  Model  using  Task  Outcomes  as  Criterion. 

If  the  task-  and  team-focused  behaviours  are  fully  mediating  between  condition  aspects  and  team  outcomes, 
the  regression  weights  from  the  conditions  to  the  team  outcomes  should  be  zero  or  near  zero.  For  this 
analysis,  PVAF  =  78%  \R2  =  .784,  F( 8,  708)  =  322.098,  p  <  0.000],  which  is  extremely  high.  This  is  an 
increase  of  approximately  2%  PVAF  (AR2  =  .024,  F(6,708)  =  12.964,/;  <  .001).  Only  two  regression  weights 
(out  of  the  six)  originating  from  the  conditions  were  non-significant  (i.e.,  organisation  characteristics  and 
team  leader),  hence,  in  our  model,  there  is  a  limited  mediating  effect  of  the  task-  and  team  processes. 
The  remaining  condition  aspects  have  a  significant  direct  effect  on  team  outcomes  over  and  above  the 
mediating  effect.  In  order  of  size  they  are:  team  characteristics,  mission  framework,  team  member,  and  task 
characteristics.  It  should  be  noticed  that  both  mission  framework  and  task  characteristics  still  have  negative 
regression  weights  (see  Figure  4-8). 


Figure  4-8:  Full  Mediating  Regression  Model  using  Team  Outcomes  as  Criterion. 


In  conclusion,  the  task  and  team  processes  act  as  mediators  on  the  relationship  between  the  condition 
components  and  the  task  and  team  outcomes  respectively,  but  there  remain  a  number  of  direct  effects  over 
and  above  the  mediating  effects.  The  mediation  effects  are  strongest  from  task  processes  to  task  outcomes 
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and  from  team-focused  behaviours  to  team  outcomes.  The  direct  effects  from  condition  to  outcomes  over 
and  above  the  effect  of  task-  and  team- focused  behaviours  are  rather  weak  (magnitude  <  .10)  and  are 
primarily  effects  of  the  conditions  on  the  team  outcomes. 

Hypotheses  9-12  were  concerned  with  the  role  of  feedback  on  task-  and  team-focused  behaviours  and 
outcomes.  All  four  of  the  hypotheses  were  supported,  providing  general  support  for  the  utility  of  feedback 
as  specified  in  the  CTEF  model: 

H9:  rfdbk_Ltao>  0  ,  r=  0.563, p<  .0001. 

Hi0:  rfdbk_Lteo  >  0,  r  =  0.652, p  <  .0001. 

Hu:  rfdbk_Ltap  >  0  ,r=  0.663,  p<  .0001. 

Hi2:  7'fdbk _ Hep  >  o,  r  =  0.676,  p<  .0001. 

4.4.4  Structural  Equation  Modelling  (SEM) 

In  the  previous  analysis  it  was  established  that  the  processes  as  specified  in  the  CTEF  model  mediate  the 
conditions  and  the  outcomes.  The  next  step  was  to  build  on  these  analyses  and  specify  in  more  detail  the 
relationship  between  the  aspects  of  the  CTEF  model  and  test  the  fit  of  that  model  against  the  data.  For  this 
analysis  we  used  the  Structural  Equation  Modelling  (SEM)  approach  which  allows  for  testing  not  only 
stepwise,  but  simultaneously  the  “added  value”  of  specific  relationships  in  a  complex  model  with  several 
independent  variables,  a  number  of  mediating  variables  and  one  or  more  dependent  variables.  It  allows  us 
to  determine  if  the  covariance  structure  in  the  data  of  a  set  of  variables  is  consistent  with  hypotheses  about 
causal  association  between  those  variables,  and  compute  the  total  effect  on  a  given  variable  by  “summing 
up”  the  effects  of  the  preceding  variables  that  are  linked  with  the  variable  under  consideration. 
Furthermore,  SEM  allows  for  finding  a  model  that  is  at  the  same  time  precise  enough  (i.e.,  close  enough  to 
the  data)  and  parsimonious  enough  (i.e.,  highlighting  the  most  important  relationships).  For  the  analyses 
we  used  the  statistical  software  AMOS  7  (together  with  SPSS  17)  (Arbuckle,  2006). 

We  started  our  analysis  with  a  very  constrained  model  (described  below)  -  i.e.,  a  model  with  fewer  paths 
than  in  the  hypothesized  model  -  and  relaxed  stepwise  the  constraints  by  allowing  more  paths. 
The  hypothesized  relationships  are  shown  in  Figure  4-9.  We  assumed  that  the  condition  components  can 
be  clustered  in  two  independent  clusters:  task  related  conditions  (TAC)  cluster  with  mission  framework 
(MF),  task  (TA),  and  organisation  (OR),  and  a  cluster  with  team  related  conditions  (TEC),  comprising 
team  leader  (TF),  team  member  (TM),  and  team  (TE).  These  two  clusters  load  independently  on  task- 
focused  behaviours  (short:  ‘task  processes’  TAP)  and  team-focused  behaviours  (short:  ‘team  processes’ 
TEP)  respectively.  The  task-focused  behaviours  load  on  task  outcomes  (TAO)  and  partly  on  team 
outcomes  (TEO);  the  team- focused  behaviours  load  on  team  outcomes  and  partly  on  task  outcomes. 
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Figure  4-9:  Model  1:  Constrained,  Starting  Model  for  Best  Fit. 

The  relationships  between  variables  are  expressed  in  path  coefficients  which  are  partial  correlations, 
showing  the  direct  effect  between  the  two  variables  under  consideration  in  the  context  of  the  other  effects. 
As  is  shown  in  Figure  4-9,  all  co-variances  (two-headed  arrows)  between  condition  aspects  are  significant 
but  one;  i.e.,  the  covariance  of  mission  framework  and  organisation.  Especially  the  strong  covariance  of 
mission  framework  and  task  characteristics  is  noteworthy  (.61).  All  path  coefficients  are  significant  except 
for  two,  mission  framework  (TAC-MF)  and  task  characteristics  (TAC-TA)  to  task-focused  behaviours 
(TAP).  This  is  consistent  with  the  analysis  described  in  the  preceding  sections.  The  overall  goodness  of  fit 
of  the  model  is  expressed  in  a  set  of  measures:  the  chi-square  test  (x2),  the  root-mean-square  residual 
(RMR),  the  Incremental  Fit  Index  (IFI),  the  Root  Mean-Square  Error  of  Approximation  (RMSEA)  and  the 
Akaike’s  Information  Criterion  (AIC).  Reference  values  in  the  search  for  a  model  with  sufficient  goodness 
of  fit  on  the  one  hand,  and  being  sufficiently  parsimonious  on  the  other  hand,  are  the  following:  regarding 
the  chi-square  test  we  strive  for  non  significance,  RMR  should  be  lower  than  .10,  IFI  greater  than  .90  and 
RMSEA  smaller  than  .08.  AIC  can  only  be  used  in  a  comparative  way  according  to  the  rule  “lower  is 
better”  (Arbuckle,  2006).  We  keep  as  an  overall  level  of  significance  for  the  path  coefficients  p  <  .05 
which  is  marked  in  the  figures  with  an  asterisk  (*).  This  starting  model  did  not  show  a  sufficient  goodness 
of  fit  (RMR  =  .148,  IFI  =  .734,  RMSEA  =  .1 14  and  AIC  =  1 195.787). 

The  next  step  was  to  search  for  a  path  which  would  best  improve  the  model:  the  modification  index  in  the 
SEM  software  suggested  the  highest  reduction  in  chi-square  value  by  freeing  a  particular  constraint  in  the 
tested  model.  The  modification  indices  suggested  introducing  a  path  from  TEP  to  TAP,  but  not  the 
converse.  This  means  that  there  is  now  an  additional  mediated  effect  of  the  team  conditions  over  the  team- 
focused  behaviours  through  task-focused  behaviours  on  both  task  outcomes  and  team  outcomes  as  shown 
in  Figure  4-10  below.  It  also  highlights  that  team-focused  behaviours  may  have  a  notable  impact  on  task- 
focused  behaviours  but  not  necessarily  the  converse. 
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Figure  4-10:  Model  2:  Improved  Model  for  Best  Fit,  Incorporating  a  Causal 
Effect  of  Team-Focused  Behaviours  on  Task-Focused  Behaviours. 


The  overall  goodness  of  fit  increased  by  introducing  the  suggested  path,  x2  dropped  from  1075.787  to 
566.426  {df=  104)  which  is  a  significant  decrease  (y2(3)  =  509.361).  The  total  x2  remained  nevertheless 
significant  which  means  that  there  is  still  room  for  improvement  of  the  model.  In  addition,  RMR  =  .10, 
IFI  =  .873,  and  RMSEA  =  .079,  which  are  borderline  satisfactory  values,  also  indicating  additional  room 
for  improvement.  AIC  decreased  from  1195.787  to  688.426;  which  is  a  change  indicating  better  fit  of  the 
model  to  the  data.  In  Model  2  (see  Figure  4-10),  all  path  coefficients  were  significant  except  for  mission 
framework;  task  characteristics  barely  achieved  significance.  The  high  value  (.75)  of  the  added  path  from 
TEP  to  TAP  is  noteworthy  because  our  original  hypotheses  did  not  specify  any  direction  for  the 
relationship  between  TAP  and  TEP.  The  observed,  strong,  directional  path  from  TEP  to  TAP  may  be 
interpreted  as  TEP  providing  extra  support  to  TAP  in  achieving  TAO  (and  TEO  to  a  lesser  extent). 

In  a  next  step  (Model  3;  see  Figure  4-11),  the  modification  indices  suggested  introducing  a  relationship 
between  TAG  and  TEP  on  the  one  hand,  and  between  TEC  and  TAP  on  the  other  hand.  This  means  that: 

1)  The  task-focused  behaviours  (TAP)  acted  now  as  a  mediator  between  team  related  conditions 
(TEC)  and  task  outcomes  (TAO);  and 

2)  That  team-focused  behaviours  (TEP)  acted  as  a  mediator  between  task-related  conditions  (TAG) 
and  team  outcomes  (TEO). 
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Figure  4-11:  Model  3:  Further  Improved  Model  for  Best  Fit,  Incorporating  Cross  Effects 
of  Conditions  Components  Selectively  on  Task-  and  Team-Focused  Behaviours. 

In  the  latter  case,  however,  only  the  relationship  between  organisation  and  TEP  were  added  as  there  was 
no  incremental  value  of  adding  paths  from  mission  framework  and  task  characteristics  to  TEP.  Similarly, 
only  a  covariance  path  between  organisation  and  team  leader  characteristics  was  suggested. 

All  added  paths  have  significant,  but  relatively  weak  coefficients.  However,  the  result  of  this  modification  is 
a  significant  increase  in  model  fit  (change  in  %2  (21)  =  73.825;  %2  (125)  =  492.601),  however  the  overall 
X2  is  still  highly  significant,  indicating  a  less  than  perfect  fit  to  the  data4.  Overall,  the  model  fit  indices 
indicate  improved  fit  to  the  data,  with  the  exception  of  RMR  (RMR  =.126,  IFI  =  .899;  RMSEA  =  .065; 
AIC  =  572.601).  RMR  increased  slightly  from  .10  to  .126.  Notably,  the  AIC  decrease,  from  688.423  to 
572.601,  is  once  more  a  change  indicating  better  fit. 

Model  3  represents  the  best  fit  to  the  data.  Further  improvements,  however,  with  weaker  effects  are 
possible  bringing  in  more  detailed  relationships.  Such  analyses  would  go  beyond  the  main  purpose  of  our 
analyses  at  this  stage  which  was  to  confirm  a  best  fit  model  based  on  strong  relationships.  An  argument 
might  be  given  to  remove  the  mission  framework  component  from  the  model:  the  relationship  with 
processes  is  non-significant,  and  a  high  correlation  is  present  with  task.  However,  conceptually  this 
component  is  an  essential  part  of  the  CTEF  model  capturing  the  environmental  characteristics.  This  issue 
will  be  discussed  in  detail  below. 

4.4.5  Differences  between  Groups 

The  survey  data  stem  from  military  personnel  with  different  backgrounds:  i.e.,  surveys  were  filled  out  by 
respondents  from  fourteen  countries;  they  worked  in  different  services;  they  identified  themselves  as  either 
leader  or  member  of  the  team  they  belonged  to;  and  they  described  diverging  types  of  missions  in  which 
they  took  part. 


4  A  model  that  would  fit  the  data  nearby  perfectly  is  called  a  saturated  model  because  it  takes  into  account  the  correlations 
between  all  variables,  or  graphically  spoken,  there  is  an  arrow  between  each  pair  of  boxes. 
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A  number  of  observations  made  during  preliminary  analyses  led  us  to  the  idea  that  different  groups  may 
have  a  different  perception  on  the  impact  of  a  given  component  on  team  effectiveness  and/or  a  different 
perception  on  the  dynamics  of  the  model  as  whole. 

In  this  section  we  analyse  these  differences  first  by  using  ANOVA  to  describe  differences  in  means  of  the 
components  of  CTEF  between  the  identified  categories  of  respondents,  and  second  by  describing  the 
differences  in  relations  between  the  components  of  CTEF.  For  this  latter  analyses  we  used  structural 
equation  modelling  (SEM,  see  Section  4.4.4).  Analyses  were  performed  once  with  the  grouping  categories 
constrained  to  be  equal,  and  once  with  a  relaxed  criterion;  i.e.,  the  grouping  categories  allowed  to  vary  to 
see  if  there  were  differences  in  path  weights  and,  eventually,  in  the  path  structures  (i.e.,  non  significant 
relations  between  the  variables  in  the  specified  model)  for  a  given  category.  The  analyses  were  conducted 
to  search  for  differences  between  countries,  services,  team  leader  vs.  team  member,  and  types  of  missions. 

4.4.5.1  Differences  between  Country  Groups 

Respondents  originated  from  14  countries  but  were  unequally  distributed  over  countries.  To  compensate  for 
that,  we  identified  three  large  categories,  called  “country  groups”  in  the  remainder,  i.e.,  United  States  (US) 
with  188  respondents,  Canada  (CA)  with  338  respondents,  and  European  Union  (EU)  with  173  respondents. 
The  respondents  from  the  EU  thus  belonged  to  twelve  countries  (see  Annex  C,  Table  C-2).  Most  of  the 
respondents  came  from  Belgium  (1 13),  the  Netherlands  (30),  and  Bulgaria  (21). 

An  ANOVA  at  the  level  of  the  components  revealed  differences  in  perceptions  between  the  three  country 
groups  as  shown  in  Table  4-7.  The  scores  reflect  the  mean  scores  per  country  group  for  each  component. 
A  higher  score  (1-5)  reflects  a  higher  magnitude  (with  the  exception  of  mission  framework  and  task). 


Table  4-7:  Effects  of  Country  Groups  on  the  CTEF  Variables. 


US 

CA 

EU 

U(2,706) 

P 

Mission  Framework 

3.29a 

3.17a 

2.91b 

13.94 

.000 

Task 

3.25a 

2.96b 

2.98b 

13.34 

.000 

Organisation 

3.29a 

3.54b 

3.59b 

10.13 

.000 

Team  Leader 

3.43a 

3.86b 

3.8  lb 

13.78 

.000 

Team  Members 

3.49a 

3.69b 

3.68b 

5.69 

.004 

Team 

3.39a 

3.64b 

3.63b 

13.41 

.000 

Task-Focused  Behaviours 

3.37a 

3.8  lb 

3.75b 

29.33 

.000 

Team-Focused  Behaviours 

3.28a 

3.69b 

3.63b 

26.03 

.000 

Task  Outcomes 

3.8  la 

4.25c 

4.02b 

24.84 

.000 

Team  Outcomes 

3.39a 

3.95b 

3.93b 

32.82 

.000 

Feedback 

4.03a 

4.4  lb 

4.53b 

11.90 

.000 

abc  Point  to  post  hoc  differences  between  two  or  three  categories,  as  tested  with  a  Scheffe  analysis  (p  <  .05). 


Table  4-7  shows  that,  for  most  of  the  components  in  the  survey,  the  respondents  from  the  US  perceived 
the  assessed  components  in  a  less  positive  way  than  the  respondents  from  Canada  and  the  EU. 
In  particular,  the  respondents  from  the  US  reported  less  task  -  and  team-focused  behaviours  than  the  other 
respondents.  They  also  reported  the  team  leader,  the  team  members,  and  the  team  to  be  less  well  equipped 
for  the  task.  Furthermore,  there  was  a  large  difference  in  team  outcomes.  Canada  and  the  EU  differed  from 
each  other  only  with  respect  to  mission  framework  and  task  outcomes. 
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The  results  can  be  seen  in  Figure  4-12.  For  each  arrow,  the  first  coefficient  (green)  represents  Canada, 
the  second  (red)  the  EU,  and  the  third  (blue)  the  US. 


Figure  4-12:  Model  3  with  Path  Coefficients  Per  Country  Group  (CA/EU/US). 

The  results  show  only  slight  differences  between  the  country  groups  in  the  paths  from  the  conditions 
(on  the  left  side  of  the  figure)  to  the  task-  and  team-focused  behaviours,  in  the  middle  of  the  figure, 
with  the  exception  of  organisation  to  team-focused  behaviours,  and  team  leader  and  team  members  to 
task-focused  behaviours.  The  association  between  organisation  and  team-focused  behaviours  is  much 
stronger  for  CA  (.18)  than  it  is  for  EU  (.10)  and  US  (.10).  The  effect  of  the  team  leader  on  task- focused 
behaviours  is  also  stronger  for  CA  than  for  the  EU  and  the  US  (.16  vs.  .11  and  .13),  whereas  for  the  EU 
the  team  members  seem  to  be  more  important  for  task-focused  behaviours  (.13  vs.  .06  and  .06). 

The  team-focused  behaviours  have  a  larger  effect  on  task-focused  behaviours  for  the  US  (.60)  than  for 
both  other  groups  (.51  and  .53). 

The  effect  of  task-focused  behaviours  on  task  outcomes  is  lowest  for  the  EU  (.42)  as  opposed  to  CA  and 
US  (.53  and  .51);  and  of  team-focused  behaviours  on  team  outcomes  (.66  vs.  .84  and  .72). 

In  summary,  there  are  differences  between  countries  but  there  is  no  consistent  pattern  throughout  the 
model.  The  analyses  confirm,  however,  that  it  is  important,  especially  in  a  multi-national  environment, 
to  be  aware  of  the  differences  in  perception  about  the  dynamics  in  command  team  effectiveness. 

4,4.5.2  Differences  between  Services 

The  respondents  were  clustered  in  three  categories  of  services:  Army  (N  =  495);  Air  Force  (N  =  134); 
and  Navy  (N  =  77).  The  Army  is  clearly  overrepresented  in  the  sample.  Moreover,  a  quick  analysis  of  the 
socio-demographics  showed  that  all  respondents  from  the  US  but  one  was  from  the  Army. 

The  main  differences  between  the  three  services  are  shown  in  Table  4-8.  The  scores  reflect  the  mean 
scores  per  service  for  each  component.  A  higher  score  (1-5)  reflects  a  higher  magnitude  (with  the 
exception  of  mission  framework  and  task). 


4-24 


RTO-TR-HFM-127 


MULTI-NATIONAL  CTEF  SURVEY  STUDY 


Table  4-8:  Effects  of  Service  on  the  CTEF  Variables. 


Army 

Air  Force 

Navy 

A(2,703) 

P 

Mission  Framework 

3.2T 

3.06 

2.89b 

7.50 

.001 

Task 

3.13a 

2.87b 

2.85b 

11.76 

.000 

Organisation 

3.45 

3.57 

3.60 

2.36 

.095 

Team  Leader 

3.68a 

3.78 

4.00b 

3.83 

.022 

Team  Members 

3.61 

3.71 

3.63 

1.08 

.342 

Team 

3.53a 

3.63 

3.71b 

4.18 

.016 

Task-Focused  Behaviours 

3.63a 

3.80b 

3.80 

5.14 

.006 

Team-Focused  Behaviours 

3.54 

3.68 

3.59 

2.32 

.099 

Task  Outcomes 

4.03 

4.16 

4.19 

2.89 

.056 

Team  Outcomes 

3.75 

3.90 

3.88 

2.03 

.133 

Feedback 

4.29 

4.48 

4.44 

2.01 

.135 

ab  Point  to  post  hoc  differences  between  two  categories,  as  tested  with  a  Scheffe  analysis  (p  <  .05). 


Table  4-8  shows  a  few  differences  between  services.  Respondents  from  the  Army  were  less  satisfied  about 
their  team  leader  than  the  respondents  from  the  Navy.  Furthermore,  the  respondents  from  the  Army  were 
less  positive  about  the  team  characteristics  and  the  task-focused  behaviours  in  their  team. 

The  SEM  analyses  with  the  three  services  as  separate  groups  resulted  in  the  coefficients  shown  in 
Figure  4-13.  For  each  arrow,  the  first  coefficient  (green)  represents  Army,  the  second  (red)  Air  Force, 
and  the  third  (blue)  Navy. 
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Concerning  the  impact  of  the  task  conditions,  for  the  Navy,  mission  framework  and  task  have  a  negative 
impact  on  the  task- focused  behaviours  (-.12  resp.  -.13),  while  the  other  services  do  not  show  a  relationship 
or  a  very  weak  one.  All  three  services  have  a  different  perception  of  the  impact  of  organisation  on  task- 
focused  behaviours,  ranging  from  .17  for  Air  Force  to  .01  for  Navy. 

Regarding  the  impact  of  team  conditions  on  team-focused  behaviours,  however,  the  Army  respondents 
perceived  a  stronger  impact  of  the  team  leader  on  task-focused  behaviours  (.34  vs.  .26  and  .29)  and  on  team- 
focused  behaviours  (.15  vs.  .09  and  .00).  Remarkable  is  the  relatively  high  impact  of  the  team  on  team- 
focused  behaviours  for  the  Air  Force  (.48  vs.  .29  and  .25)  and  of  the  team  on  task-focused  behaviours  for  the 
Navy  (.38  vs.  .13  and  .10). 

The  Army  showed  the  strongest  effects  from  team-focused  behaviours  on  task-focused  behaviours 
(.58  vs.  .51  and  .46)  and  from  task-focused  behaviours  on  task  outcomes  (.51  vs.  .46  and  .40)  and  of  team- 
focused  behaviours  on  team  outcomes  (.76  vs.  .59  and  .69). 

In  summary,  the  respondents  from  the  Army  perceived  stronger  effects  from  the  team  leader  than  the  other 
services,  whereas  the  other  services  saw  more  beneficial  effects  from  the  team.  Also,  both  task  and  team 
outcomes  were  better  explained  by  task-  and  team-focused  behaviours  in  the  Army  sample  than  for  the 
other  services. 

4.4.5.3  Differences  between  Team  Leaders  and  Team  Members 

Of  all  respondents  265  identified  themselves  as  the  leader  of  the  team  they  were  referring  to  and 
452  stated  they  were  a  member  of  the  team.  The  differences  in  perception  between  the  two  categories  are 
shown  in  Table  4-9.  The  scores  reflect  the  mean  scores  for  either  leader  or  member  position  in  the  team 
for  each  component.  A  higher  score  (1-5)  reflects  a  higher  magnitude  (with  the  exception  of  mission 
framework  and  task). 


Table  4-9:  Effects  of  Position  in  the  Team  on  the  CTEF  Variables. 


Leader 

Member 

^(1,715) 

P 

Mission  Framework 

3.16 

3.13 

.29 

.591 

Task 

3.04 

3.06 

.12 

.730 

Organisation 

3.56 

3.44 

4.46 

.035 

Team  Leader 

3.97 

3.59 

27.80 

.000 

Team  Members 

3.69 

3.60 

2.71 

TOO 

Team 

3.62 

3.54 

2.68 

.102 

Task-Focused  Behaviours 

3.85 

3.57 

29.08 

.000 

Team-Focused  Behaviours 

3.77 

3.45 

38.12 

.000 

Task  Outcomes 

4.24 

3.97 

23.14 

.000 

Team  Outcomes 

4.01 

3.67 

27.81 

.000 

Feedback 

4.67 

4.15 

42.33 

.000 

Table  4-9  shows  that  team  leaders  were  far  more  positive  than  team  members  with  respect  to  both  the  task 
and  team-focused  behaviours,  the  task  and  team  outcomes,  and  the  feedback  in  their  teams.  In  the  conditions, 
there  are  only  two  components  for  which  team  leaders  differed  from  team  members.  The  leaders  were  more 
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positive  about  the  qualities  of  the  team  leaders  (being  themselves)  than  the  team  members,  and  also  more 
positive  about  the  organisation. 

The  SEM  analyses  revealed  the  following  paths  coefficients  as  shown  in  Figure  4-14.  The  first  coefficient 
(red)  represents  the  leader  and  the  second  coefficient  (blue)  the  team  member. 


Figure  4-14:  Model  3  with  Path  Coefficients  for  Team  Leader  and  Team  Member  as  Groups. 


Among  the  impacts  of  task  conditions  on  task-focused  behaviours,  we  observe  only  one  noteworthy 
difference;  i.e.,  for  team  leaders,  mission  framework  has  an  impact  on  the  task-focused  behaviours  while 
team  members  do  not  see  any  association  at  all  (.12  vs.  .01). 

Among  the  impacts  of  team  conditions  on  task-  and  team-focused  behaviours,  we  observe  a  divergence 
between  team  leaders  and  team  members  with  respect  to  two  relationships:  team  leaders  believe  to  have  a 
stronger  impact  on  task-focused  behaviours  (.21  vs.  .10)  and  on  team-focused  behaviours  (.34  vs.  .27), 
and  team  members  believe  that  team  characteristics  have  more  impact  on  task-focused  behaviours 
(.21  vs.  .08)  and  on  team- focused  behaviours  (.37  vs.  .  22). 

As  a  partial  conclusion,  we  can  say  that  team  leaders  believe  that  their  qualities  are  most  important  for  the 
quality  of  the  task-  and  team-focused  behaviours,  whereas  the  team  members  think  the  team  is  more 
important  for  the  quality  of  both  task-  and  team-focused  behaviours. 

4.4.5.4  Differences  between  Task  Types 

Formulated  as  an  open  ended  question,  respondents  reported  the  kind  of  mission  their  team  took  part  in. 
Based  on  the  descriptions  of  the  respondents,  we  classified  their  answers  in  one  of  four  categories:  battle/ 
combat  missions  (N  =  283),  training  missions  (N  =  110);  peacekeeping/stability  missions  (N  =  67), 
and  support  missions  (N  =  245). 
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Table  4-10  shows  the  differences  in  perception  of  respondents  depending  on  the  kind  of  missions  considered. 
The  scores  reflect  the  mean  scores  per  mission  type  for  each  component.  A  higher  score  (1-5)  reflects  a 
higher  magnitude  (with  the  exception  of  mission  framework  and  task). 


Table  4-10:  Effects  of  Mission  Type  on  the  CTEF  Variables. 


Battle/Combat 

Training 

PK/Stability 

Support 

F(3,714) 

P 

Mission  Framework 

3.15a 

3.35b 

3.14a 

3.06a 

3.48 

0.008 

Task 

3.00a 

3.44b 

2.93a 

2.96a 

11.41 

0.000 

Organisation 

3.60a 

3.14b 

3.38c 

3.54abc 

9.13 

0.000 

Team  Feader 

3.87a 

3.24b 

3.74a 

3.82a 

11.20 

0.000 

Team  Members 

3.63ab 

3.53a 

3.54ab 

3.70b 

1.52 

0.195 

Team 

3.63a 

3.38b 

3.44b 

3.63a 

5.06 

0.001 

Task-Focused  Behaviours 

3.74a 

3.37b 

3.62a 

3.76a 

7.95 

0.000 

Team-Focused  Behaviours 

3.62ac 

3. 1 6b 

3.50a 

3.7  lc 

14.30 

0.000 

Task  Outcomes 

4. 1 5a 

3.75b 

4.05a 

4. 14a 

7.64 

0.000 

Team  Outcomes 

3.89ac 

3.2  lb 

3.74a 

3.96c 

18.02 

0.000 

Feedback 

4.38a 

4.02b 

4.34a 

4.48a 

5.11 

0.000 

abc  Point  to  post  hoc  differences  between  two  or  three  categories,  as  tested  with  a  Scheffe  analysis  ip  <  .05). 


Table  4-10  highlights  differences  between  the  four  types  of  missions,  with  the  training  missions  often 
standing  alone  as  differing  from  the  combat,  stability,  and  support  mission  types.  Respondents  describing 
training  missions  viewed  the  mission  framework,  task  characteristics  as  more  difficult  and  generally  with 
less  organisational  support  than  the  other  three  mission  types.  These  respondents  also  consistently 
evaluated  task-related  and  team-related  process  behaviours  and  outcomes  lower  than  the  other  mission 
types.  This  generally  reflects  a  view  that  the  training  types  of  mission  are  consistently  more  difficult  than 
other  mission  types.  Also  of  note  is  that  respondents  describing  support  missions  evaluated  their  team- 
related  behaviours  and  outcomes  more  favourably  than  peacekeeping/stability  missions  (in  addition  to 
training  missions).  This  may  reflect  the  premium  and  focus  on  coordination  for  support  and  logistics  types 
of  missions. 

The  SEM  analysis  with  the  four  types  of  missions  as  separate  groups  shows  the  following  results 
(see  Figure  4-15).  For  each  arrow,  the  first  coefficient  (green)  represents  battle/command,  the  second  (red) 
training,  the  third  (blue)  peacekeeping/stability,  and  the  fourth  (yellow)  support. 


4-28 


RTO-TR-HFM-127 


MULTI-NATIONAL  CTEF  SURVEY  STUDY 


Figure  4-15:  Model  3  with  Path  Coefficients  for  Task  Types  as  Groups. 

The  impact  of  the  team  leader  on  the  team- focused  behaviours  is  highest  in  training  (.43)  while  it  is  lowest  in 
combat  (.19).  Similarly,  team  members  impact  most  in  training  (.26)  and  combat  (.25)  and  least  in  peace 
operations  (.12).  Conversely,  the  team  as  a  whole  is  most  important  in  peace  operations  (.41)  and  in  support 
(.40)  as  opposed  to  combat  (.27)  and  training  (.26). 

The  organisation  characteristics  contribute  positively  to  team-focused  behaviours  in  all  missions,  except 
for  Support  missions  (-.04). 

The  impact  of  people’s  competencies  on  the  task-focused  behaviours  depends  on  the  focus.  The  team 
leader  contributes  most  in  combat  (.17)  and  training  (.18)  and  least  in  both  other  mission  types  (.08), 
whereas  team  members  have  only  a  significant  contribution  in  support  (.17  vs.  .00  to  .02).  The  impact  of 
the  team  characteristics  is  highest  in  combat  (.23)  and  least  in  training  (.05). 

The  impact  of  team-focused  behaviours  on  task-focused  behaviours  is  very  strong  and  highest  for  peace 
operations  (.68)  but  lowest  although  quite  significant  in  combat  (.51). 

The  direct  impact  of  task- focused  behaviours  on  both  task  outcomes  and  team  outcomes  is  relatively  low  for 
training  missions  (.41  resp.  .01)  as  compared  to  other  types  of  missions  (.54,  .47,  .47,  resp.  .21,  .29,  .20). 
On  the  other  hand,  in  training  missions  the  direct  effects  of  team-focused  behaviours  on  task  outcomes  and 
team  outcomes  are  higher  than  for  the  other  types  of  missions  (.31  vs.  .21,  .18,  .26,  resp.  .89  vs.  .68,  .59,  .66). 

It  can  be  concluded  that  the  outcomes  of  training  missions  are  affected  by  different  factors  than  other 
missions.  Furthermore,  the  role  of  the  leader  and  the  team  members  is  most  important  in  these  missions. 

4.4.6  Participants’  Feedback  on  the  CTEF  Instrument 

Upon  completion  of  the  CTEF  survey,  participants  were  provided  with  a  number  of  questions  addressing 
the  utility  and  usability  of  the  CTEF  instrument.  Specifically,  we  were  interested  in  whether  respondents 
believed  that  the  CTEF  instrument  addressed  key  aspects  of  team  effectiveness  as  viewed  by  military 
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personnel  in  command  teams.  Additionally,  we  wished  to  ensure  that  the  questions  and  rating  scales  were 
clear  and  useful.  Moreover,  one  of  the  goals  of  the  CTEF  instrument  was  to  lead  respondents  to  learn 
more  about  the  components  of  team  effectiveness.  The  responses  to  these  questions  are  summarized  in 
Table  4-11. 


Table  4-1 1 :  Respondent  Feedback  on  the  Utility  and  Usability  of  CTEF  Instrument. 


Questions  in  Survey 

Completely 

Disagree 

Disagree 

Neutral 

Agree 

Completely 

Agree 

The  right  aspects  of  team 
effectiveness  are  being 
addressed 

8 

47 

158 

462 

39 

The  questions  are  clear 

3 

72 

120 

468 

51 

The  distinction  between  level 
and  impact  is  clear 

5 

70 

161 

434 

44 

The  distinction  between  level 
and  impact  is  useful 

3 

40 

187 

431 

53 

I  learned  about  improving  team 
effectiveness 

35 

153 

269 

221 

36 

Note:  N  =  714,  Missing  =  4. 


Encouragingly,  approximately  70%  of  respondents  either  “agreed”  or  “completely  agreed”  that  the 
instrument  addresses  the  important  aspects  of  command  teams;  very  few  participants  either  “disagreed”  or 
“completely  disagreed”  (N  =  55;  7.66%).  In  addition,  519  (72.3%)  of  respondents  either  “agreed”  or 
“completely  agreed”  that  the  questions  were  clear.  Many  respondents  found  the  distinction  between  level 
and  impact  to  be  useful  but  a  sufficient  number  of  respondents  did  not  find  the  distinction  to  be  clear. 
Taken  together,  these  findings  provide  some  insight  into  why,  perhaps,  the  distinction  between  magnitude 
(i.e.,  level  or  degree)  and  impact,  while  conceptually  clear  to  researchers,  may  not  be  clear  to  and 
practically  useful  for  the  end-user  population.  Finally,  an  encouraging  percentage  (35%)  of  command  team 
members  and  leaders  felt  that  simply  working  through  the  survey  made  them  more  aware  of  team 
effectiveness  than  they  were  before  conducting  the  survey. 

From  early  feedback  in  operational  exercises,  we  learned  that  the  originally  proposed  CTEF  instrument 
was  viewed  as  being  too  long.  In  subsequent  revisions  we  attempted  to  address  this  issue,  and  as  a  result 
we  also  included  a  question  to  gauge  the  extent  to  which  the  length  of  the  instrument  used  in  the  survey 
was  acceptable  to  respondents.  The  results  are  provided  in  Table  4-12.  Overall,  576  (80%)  of  participants 
who  completed  the  survey  found  the  length  to  be  “acceptable”  or  better.  Of  course,  some  proportion  of 
respondents  who  initiated  the  survey  may  have  not  completed  it  because  it  was  longer  than  they  expected. 
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Table  4-12:  Respondents  on  the  Length  of  the  Questionnaire. 


Frequency 

Percent 

Much  too  long 

20 

2.8 

A  little  too  long 

118 

16.4 

Acceptable 

464 

64.6 

A  little  too  short 

93 

13.0 

Much  too  short 

19 

2.6 

Total 

714 

99.4 

System  Missing 

4 

0.6 

The  final  feedback  question  was  important  in  establishing  if  our  respondents  would  be  willing  to  actually  use 
the  CTEF  in  educational  establishments  (e.g.,  staff  colleges,  military  academies),  training  or  exercise 
environments,  or  operational  settings.  For  this  item,  participants  checked  any  of  the  three  settings  they  would 
consider  (which  is  why  the  data  sum  to  more  than  the  718  participants),  or  else  responded  “not  at  all”. 
The  results  are  presented  in  Table  4-13  where  we  provide  a  breakdown  of  the  data  by  nationality  of  the 
respondent.  Encouragingly,  an  overwhelming  proportion  of  respondents  thought  that  they  would  indeed  use 
the  CTEF  in  educational,  training,  and  even  operational  settings.  Thus,  the  results  of  this  question,  together 
with  the  responses  provided  above,  provide  confidence  that  CTEF  is  an  instrument  that  resonates  well  with 
command  team  leaders  and  members,  and  is  of  practical  utility  for  a  range  of  military  applications. 


Table  4-13:  Respondents  on  Willingness  to  Use  CTEF. 


Country 

Educational 

Settings 

Training 

Settings 

Operational 

Settings 

Not  at  All 

Total 

Belgium 

13 

24 

37 

39 

113 

Bulgaria 

9 

2 

16 

27 

Canada 

144 

199 

136 

55 

534 

Czech  Republic 

1 

1 

Denmark 

1 

1 

France 

1 

1 

1 

3 

Germany 

1 

1 

Netherlands 

9 

13 

13 

5 

40 

Poland 

1 

1 

Russia 

1 

1 

Spain 

1 

1 

United  Kingdom 

1 

1 

United  States 

86 

113 

72 

43 

314 

Total 

263 

353 

277 

145 

1038 
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4.5  DISCUSSION 

In  this  chapter  we  sought  to  address  the  first  two  goals  of  the  survey  study;  i.e.,: 

1)  To  examine  the  conceptual  consistency  of  the  framework  and  the  psychometric  properties  of  the 
instrument;  and 

2)  To  determine  the  applicability  of  the  framework  and  the  instrument  for  command  teams  in 
different  NATO  nations  and  diverse  operational  settings. 

Through  extensive  analyses,  we  found  the  psychometric  properties  of  the  CTEF  instrument  to  be  generally 
acceptable.  The  items  of  the  CTEF  instrument  were  intended  to  be  summative  measures  of  the  aspects  of 
the  CTEF  model.  Although  the  items  associated  with  the  condition  components  generally  exhibited  these 
properties,  the  processes  and  outcome  items  did  not  exhibit  these  properties  in  all  cases.  This  is  discussed 
in  more  detail  in  Chapter  5. 

Through  the  hypotheses  proposed  and  their  associated  results,  we  found  that  the  CTEF  instrument  exhibits 
conceptual  consistency  with  existing  theory  and  empirical  research  on  team  effectiveness.  The  observed 
pattern  of  relationships  reproduces  the  mediational  role  of  processes,  as  well  as  the  distinction  between 
task- focused  behaviours  and  team-focused  behaviours  as  posited  by  McIntyre  and  Salas  (1995). 
team  related  condition  components  (team  leader,  team  members,  team)  were  more  strongly  associated  with 
team-focused  behaviours  as  well  as  team  outcomes.  A  similar  pattern  of  relationships  was  observed  with 
task  related  components  (mission  framework,  task  characteristics,  organisation),  although  the  relationships 
between  task-related  components  and  task-focused  behaviours  were  notably  weaker  than  expected. 
Revisions  of  the  CTEF  instrument  with  regard  to  the  task  related  condition  components  factors  should 
substantially  address  this  weakness. 

In  order  to  address  the  second  goal  of  the  survey  study,  we  analysed  the  data  by  several  backgrounds. 
We  found  notable  differences  between  categories  of  the  grouping  variables.  First,  there  were  differences 
between  countries.  The  respondents  from  the  US  in  general  showed  lower  magnitude  judgments  on  the 
components  of  the  model  than  respondents  from  Canada  and  the  EU.  There  were  also  slight  differences  in 
the  paths  in  the  model.  The  results  show  only  slight  differences  between  the  countries  in  the  paths  from  the 
conditions  to  the  task-  and  team-focused  behaviours,  with  the  exception  of  organisation  to  team-focused 
behaviours,  and  team  leader  and  team  members  to  task-focused  behaviours.  These  differential  effects 
suggest  that  for  Canada  the  organisation  seems  to  be  more  important,  for  the  US  the  team  leader,  and  for 
the  EU  the  team  members.  Second,  there  were  differences  between  services.  The  respondents  from  the 
Army  were  less  positive  about  their  team  leaders  and  team  characteristics  than  the  respondents  from  the 
Navy.  There  were  also  differential  effects  of  the  paths  in  the  CTEF  model.  The  Army  respondents  saw  a 
stronger  impact  of  the  leader  on  the  processes,  whereas  the  Air  Force  and  the  Navy  accentuated  the 
influence  of  the  team.  Third,  there  were  differences  between  team  leaders  and  team  members  in  their 
perceptions.  In  general,  team  leaders  were  more  positive  about  the  processes,  the  outcomes,  and  the 
feedback  than  team  members. 

Finally,  we  collected  considerable  feedback  from  respondents  about  the  CTEF  instrument.  We  collected 
responses  from  military  leaders  and  members  of  command  teams  in  a  variety  of  settings  and  countries. 
Responses  to  the  feedback  questions  were  consistent  across  countries  and  settings.  Respondents  generally 
found  the  CTEF  instrument  to  be  clear,  easy  to  understand,  and  useful.  Approximately  80%  of  respondents 
also  indicated  that  they  would  be  willing  to  use  the  CTEF  instrument  in  educational,  training, 
or  operational  settings. 
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Chapter  5  -  DISCUSSION 

5.1  OVERVIEW  OF  RESULTS 

The  primary  goals  of  the  study  included: 

•  To  examine  the  conceptual  consistency  of  the  model  and  the  psychometric  properties  of  the 
instrument; 

•  To  determine  the  applicability  of  the  model  and  the  instrument  for  command  teams  in  different 
NATO  nations  and  diverse  operational  settings;  and 

•  To  revise,  consolidate,  and  update  the  model  and  instrument  based  on  the  findings  of  the  study. 

In  Chapter  4  we  examined  the  conceptual  consistency  of  the  CTEF  model  with  a  model  fit  test,  and  we 
explored  the  psychometric  properties  of  the  instrument  based  on  the  survey.  The  size  of  the  inter-item 
correlations  and  also  the  correlations  between  the  scales  that  represented  the  magnitude,  on  the  one  hand, 
and  the  impact,  on  the  other,  is  generally  large.  These  are  indications  that  we  could  possibly  reduce  the 
number  of  items  and  scales  in  the  survey. 

From  the  analyses  for  each  of  the  aspects  or  components  we  can  draw  the  following  conclusions: 

•  The  correlations  among  the  items  in  mission  framework  did  not  exceed  .35,  indicating  that  the 
respondents  were  able  to  distinguish  between  items  that  were  conceptually  different. 

•  For  the  task  characteristics,  most  items  did  not  correlate  very  highly,  with  the  exception  of  task 
ambiguity  and  task  instability  (.61).  In  the  latter  case  it  is  not  clear  if  this  correlation  reflects  a 
mere  co-occurrence  or  a  causal  relation;  e.g.,  instability  leads  to  ambiguity. 

•  All  inter-correlations  for  the  organisation  characteristics  items  were  in  the  range  of  .17  to  .43. 
Hence  the  various  aspects  should  be  retained. 

•  The  three  items  measuring  the  team  leader  correlated  highly.  In  particular,  the  items  team  leader 
skills  and  team  leader  knowledge  were  hard  to  distinguish.  These  items  could  be  replaced  with 
one:  the  leader’s  competencies. 

•  The  three  items  representing  the  team  members  also  correlated  highly.  Here  also  the  items  of  team 
member  skills  and  team  member  knowledge  may  be  replaced  with  one:  team  member’s 
competencies. 

•  The  items  representing  the  team  did  not  correlate  very  highly.  Hence,  these  items  should  be 
retained,  except  for  team  size. 

•  The  items  representing  task-focused  behaviours  did  correlate  highly.  In  a  number  of  cases  the 
correlations  were  larger  than  .70,  suggesting  that  we  could  reduce  the  number  of  items  (i.e.,  retain 
the  Fevel  2  items,  such  as  managing,  planning,  etc.). 

•  The  same  applies  to  the  team-focused  behaviours.  They  also  correlated  very  highly  for  a  number 
of  items.  Here  also,  it  can  be  recommended  to  look  for  clusters  of  items  that  can  be  aggregated. 

•  The  items  representing  the  task  outcomes  correlated  higher  than  .50,  although  the  items  clearly  try 
to  measure  separate  concepts.  Hence,  we  keep  them  as  such. 

•  The  team  outcome  items  all  correlated  higher  than  .60,  suggesting  that  we  may  be  able  to  reduce 
the  number  of  items  to  one  or  two. 

•  The  feedback  items  all  correlated  above  .70,  suggesting  the  number  of  items  here  could  be 
reduced  to  one  or  two  “overarching”  items. 
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The  overall  correlation  between  the  magnitude  and  the  impact  scales  was  .72,  suggesting  considerable 
overlap.  Further  inspection  of  the  data  showed  that  the  high  correlation  between  scales  was  predominant  in 
the  task-  and  team-focused  behaviours  and  in  the  conditions  representing  the  team,  the  leader,  and  the 
members.  The  correlations  were  lower  for  the  mission  framework,  task  characteristics,  and  organisation 
characteristics.  Nevertheless,  the  results  taken  together  suggest  that  it  is  not  necessary  to  retain  both  the 
magnitude  and  the  impact  scales. 

On  the  basis  of  the  high  inter-item  correlations  and  the  high  correlations  between  the  magnitude  and  the 
impact  scales,  we  will  provide  recommendations  for  reducing  the  number  of  items  in  the  CTEF  instrument 
(see  Section  5.3).  But  before  that,  it  is  important  to  verify  that  the  high  correlations  evident  in  the  survey 
study  are  also  evident  when  data  are  collected  during  an  actual  operation.  The  principle  concern  here  is 
that  the  correlations  evident  in  the  survey  study  could  have  been  susceptible  to  a  form  of  hindsight  bias1. 
We  will  discuss  this  phenomenon  in  more  detail  in  Section  5.2,  where  we  re-analyze  the  data  collected  in 
exercise  Joint  Caribbean  Lion  (i.e.,  JCL2  and  JCL3). 

We  also  tested  an  extensive  set  of  hypotheses  that  could  be  derived  from  the  CTEF  model.  In  general, 
most  hypotheses  held.  First,  four  of  the  conditions  correlated  positively  with  the  task-focused  behaviours 
and  the  team-focused  behaviours:  Organisation,  team,  team  leader,  and  team  members  all  had  positive 
effects  on  task-focused  and  team-focused  behaviours.  However,  contrary  to  what  we  originally  expected, 
we  found  no  correlations  between  the  mission  framework  and  task-  and  team-focused  behaviours  and  even 
a  slightly  negative  relationship  between  task  characteristics  and  task-  and  team-focused  behaviours.  These 
latter  results  mean  that  more  complex,  ambiguous,  or  difficult  tasks  result  in  better  task  and  team 
activities.  A  further  analysis  showed  that  there  is  a  notable  difference  between  English  and  non-English 
questionnaires  with  respect  to  this  relationship.  The  negative  relationships  held  only  for  the  English 
questionnaires  and  not  for  non- English  respondents  (i.e.,  beta- values  of  -.18  vs.  .05  with  task- focused 
behaviours  as  the  dependent  variable  and  beta-values  of  -.12  vs.  .05  with  team- focused  behaviours  as  the 
dependent  variable). 

Second,  task-focused  behaviours  and  team-focused  behaviours  had  positive  effects  on  both  task  outcomes 
and  team  outcomes,  although  task-focused  behaviours  were  more  important  for  task  outcomes  than  team- 
focused  behaviours,  and  team- focused  behaviours  had  a  larger  effect  on  team  outcomes  than  task-focused 
behaviours.  These  effects  and  differences  in  the  size  of  the  effects  were  all  as  hypothesized.  One  concern, 
however,  was  the  magnitude  of  the  effects;  the  correlations  were  all  in  the  range  of  .65  or  higher.  The  size 
of  the  correlations  suggests  that  the  concepts  may  have  been  subject  to  a  halo2  effect  and  thus  again  related 
to  the  retrospective  nature  of  the  survey  study.  This  possibility  will  also  be  addressed  in  Section  5.2  when 
we  re-examine  the  data  from  Joint  Caribbean  Lion. 

Third,  the  feedback  was  positively  correlated  both  with  the  task  and  team  outcomes,  and  with  the  task-  and 
team-focused  behaviours.  This  means  that  a  better  review  process  leads  to  better  processes  and  better 
outcomes.  We  saw  no  confirmation  of  the  more  dynamic  correcting  function  of  the  feedback  loops. 
According  to  this  idea  the  feedback  loops  should  show  a  negative  relationship  with  team  and  task 
outcomes:  e.g.,  the  worse  the  outcomes  the  more  attention  for  the  improvement  of  the  conditions  and/or 
the  processes.  However,  if  we  would  have  liked  to  test  that  we  should  have  used  a  more  dynamic  approach 
in  testing.  From  the  survey  data,  the  only  thing  that  we  can  conclude  is  that  having  better  feedback  loops 
helps  the  team  to  improve  their  processes  and  their  outcomes. 


1  Hindsight  bias  means  that  people,  when  looking  backward,  see  a  stronger  “consistent”  relationship  between  aspects  than  there 
was  objectively  present  at  the  very  moment  of  the  events. 

2 

Halo  effect  means  that  the  “value”  of  one  characteristic  is  erroneously  seen  “in  the  light  of’  the  value  of  another  one. 
This  leads  to  inflated  associations  or  correlations,  e.g.,  it  has  repeatedly  been  proven  that  physically  attractive  people  are 
believed  to  be  more  intelligent  than  less  attractive  people. 
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Fourth,  the  model  as  a  whole  was  tested  with  path  model  analysis  and  with  structural  equation  modelling. 
A  closer  look  with  path  analysis  also  showed  that  the  task-  and  team-focused  behaviours  mediated  to  a 
large  extent  the  relationship  between  the  conditions  and  task  and  team  outcomes.  However,  there  were  a 
few  additional  direct  relationships  between  the  conditions  and  the  outcomes.  The  structural  equation 
Model  3  showed  a  reasonable  fit  to  the  data;  therefore  we  can  conclude  that  our  model  holds  conceptually 
throughout  all  applications. 

In  order  to  address  the  second  goal  of  the  survey  study,  we  analysed  the  data  from  several  perspectives 
(backgrounds  of  respondents)  in  order  to  see  if  there  were  differences.  Although  we  found  many 
interesting  differences,  we  can  conclude  that  the  model  holds  for  all  categories  of  respondents  that  we 
distinguished.  Furthermore,  we  collected  responses  from  military  leaders  and  members  of  command  teams 
about  the  model  and  the  instrument.  Respondents  generally  found  the  CTEF  instrument  to  be  clear,  easy  to 
understand,  and  useful.  Approximately  80%  of  all  respondents  also  indicated  that  they  would  be  willing  to 
use  the  CTEF  instrument  in  educational,  training,  or  operational  settings. 


5.2  COMPARISON  OF  SURVEY  DATA  WITH  JOINT  CARIBBEAN  LION 

As  discussed  previously,  the  size  of  both  the  inter-item  correlations  and  the  correlations  between  different 
aspects  or  components  in  the  survey  could  have  been  influenced  by  the  fact  that  the  respondents  had  to 
refer  to  a  mission  or  exercise  which  they  had  been  part  of  some  time  ago.  Therefore,  the  size  of  the 
correlations  could  have  been  biased  by  both  halo-effects  and  by  leniency  effects  (e.g.,  Cooper,  1981; 
Harris  and  Schaubroeck,  1988).  A  halo  effect  implies  that  one  general  underlying  factor  accounts  for  high 
correlations.  A  potential  candidate  for  such  an  underlying  factor  is  the  general  feeling  that  remains  after 
the  mission.  Leniency  implies  that  in  hindsight  respondents  may  see  the  events  in  a  more  positive  light 
than  might  have  been  justified  by  their  real  experiences  during  the  operation.  Although  both  effects  have 
been  known  to  be  a  problem  of  surveys  in  general  (mono-method  bias  and  response  tendencies), 
we  assume  that  these  effects  would  be  reduced  when  respondents  fill  out  a  questionnaire  about  a  mission 
or  exercise  that  is  going  on  at  present. 

In  order  to  test  our  assumptions,  we  compared  the  survey-data  with  the  data  from  the  second  and  third 
administrations  of  Joint  Caribbean  Lion.  We  did  not  use  the  first  administration  of  Joint  Caribbean  Lion, 
because  in  that  administration  we  used  the  long  version  of  the  instrument  and  prior  to  changing  the  reading 
level  of  the  items.  We  merged  the  data  from  both  administrations  two  and  three  into  one  dataset  in  order  to 
have  enough  respondents  to  test  the  relationships  in  the  model.  First,  we  left  out  the  four  observers  of  the 
exercise  in  order  to  have  only  team  leaders  and  team  members.  Second,  we  checked  if  there  was  not  too 
much  overlap  in  respondents.  Only  8  respondents  out  of  58  had  filled  out  the  questionnaire  at  both  points 
in  time.  Because  of  the  relatively  little  amount  of  overlap  we  decided  to  proceed  with  all  remaining 
entries. 

In  Table  5-1  we  compared  the  sizes  of  the  inter-item  correlations  in  the  survey  with  those  in  Joint 
Caribbean  Lion.  For  each  component  in  the  model  we  noted  the  minimum,  the  maximum,  and  the  median 
of  all  inter-item  correlations.  Table  5-1  shows  that  the  medians  of  the  inter-item  correlations  of  the  Joint 
Caribbean  Lion  data  were  considerably  lower  for  most  of  the  aspects.  The  largest  reductions  in  median 
inter-item  correlations  were  found  in  the  aspects  organisation,  team  leader,  team  members,  task-focused 
behaviours,  and  task  outcomes.  The  inter-item  correlations  within  the  aspect  task  characteristics  were  in  a 
narrower  range  in  the  Joint  Caribbean  Lion  data.  The  inter-item  correlations  for  the  team-focused 
behaviours  and  for  the  team  outcomes  were  slightly  lower  for  the  Joint  Caribbean  Lion  data  than  for  the 
survey  data.  There  is  only  one  component  in  which  the  inter-item  correlations  were  larger  for  the  Joint 
Caribbean  Lion  data  than  for  the  survey  data:  the  aspect  team.  Therefore,  we  can  conclude  that  overall  the 
survey  data  may  have  been  affected  by  halo  effects  that  result  from  hindsight  bias,  and  that  this  effect  is 
larger  for  the  task-focused  behaviours  and  task  outcomes  than  for  the  team-focused  behaviours  and  the 
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team  outcomes.  However,  the  inter-item  correlations  for  Joint  Caribbean  Lion  still  were  quite  high. 
This  suggests  that  it  is  justified  to  strive  for  a  reduction  in  the  number  of  items  in  the  survey. 

Table  5-1:  A  Comparison  of  the  Inter-Item  Correlation  Coefficients 
of  the  Survey  with  the  Data  from  Joint  Caribbean  Lion. 


Survey  Joint  Caribbean  Lion 


Minimum 

Maximum 

Median 

Minimum 

Maximum 

Median 

Mission  Framework 

.13 

.35 

.28 

.11 

.34 

.24 

Task 

-.00 

.61 

.12 

.19 

.41 

.27 

Organisation 

.17 

.43 

.30 

.10 

.34 

.17 

Team  Leader 

.64 

.75 

.68 

.51 

.66 

.57 

Team  Members 

.53 

.73 

.53 

.41 

.49 

.45 

Team 

.06 

.41 

.28 

.19 

.50 

.42 

Task-Focused  Behaviours 

.31 

.71 

.52 

.17 

.70 

.41 

Team-Focused  Behaviours 

.32 

.69 

.47 

.15 

.71 

.44 

Task  Outcomes 

.54 

.63 

.60 

.46 

.58 

.50 

Team  Outcomes 

.61 

.76 

.67 

.53 

.71 

.62 

We  hypothesized  that  not  only  the  inter-item  correlations  would  have  been  affected  by  the  halo  effects, 
but  also  the  correlations  between  the  aspects  in  the  model.  In  order  to  test  this,  we  compared  the 
standardised  regression  coefficients  (beta- values  in  the  table)  of  the  survey  data  with  the  data  from  Joint 
Caribbean  Lion  (see  Table  5-2). 
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Table  5-2:  Beta  Coefficients  and  R2  Values  for  the  Data  from  the  Survey  and  Joint  Caribbean  Lion. 


Survey 

Beta- 

Values 

Survey 

R2 

JCL 

Beta- 

Values 

JCL 

R2 

Conditions  -> 

Mission  Framework 

.02 

.37 

.10 

.16 

Task  Outcomes 

Task 

-.10 

-.02 

Organisation 

.14 

.34 

Team  Leader 

.33 

-.03 

Team  Member 

.11 

-.07 

Team 

.20 

.14 

Conditions  -> 

Mission  Framework 

-.06 

.53 

.10 

.45 

Team  Outcomes 

Task 

-.09 

-.07 

Organisation 

.12 

.09 

Team  Leader 

.28 

.22 

Team  Member 

.21 

.05 

Team 

.31 

.45 

Conditions  -> 

Mission  Framework 

.04 

.55 

-.00 

.57 

Task-Focused 

Task 

-.10 

-.03 

Behaviours 

Organisation 

.15 

.20 

Team  Leader 

.31 

.21 

Team  Member 

.18 

.10 

Team 

.31 

.45 

Conditions  -> 

Mission  Framework 

.00 

.51 

.05 

.51 

Team-Focused 

Task 

-.04 

.00 

Behaviours 

Organisation 

.12 

.13 

Team  Leader 

.32 

.26 

Team  Member 

.19 

.07 

Team 

.29 

.43 

Behaviours  -> 

Task-Focused  Behaviours 

.51 

.52 

.46 

.19 

Task  Outcomes 

Team-Focused  Behaviours 

.25 

-.03 

Behaviours  -> 

Task-Focused  Behaviours 

.16 

.76 

.06 

.61 

Team  Outcomes 

Team-Focused  Behaviours 

.74 

.73 

Table  5-2  shows  some  notable  differences  between  the  data  sets: 

•  First,  contrary  to  the  survey  data,  the  task  outcomes  in  Joint  Caribbean  Lion  were  not  well 
explained,  neither  by  the  conditions  nor  by  the  behaviours.  In  Joint  Caribbean  Lion,  only  the 
organisation  characteristics  and  the  task- focused  behaviours  had  some  effect  on  the  task  outcomes. 

•  Second,  the  explained  variance  (PVAF)  of  the  team  outcomes  in  Joint  Caribbean  Lion  was 
somewhat  lower  than  the  explained  variance  of  the  team  outcomes  in  the  survey.  However, 
the  explained  variance  of  the  task-  and  team-focused  behaviours  by  the  conditions  was  about  the 
same  for  both  datasets. 

•  Third,  team  showed  larger  beta-values  in  Joint  Caribbean  Lion  than  in  the  survey,  whereas  in  the 
survey  the  leader  was  the  dominant  factor  for  the  explained  variance  of  task-  and  team-focused 
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behaviours  as  well  as  task  and  team  outcomes.  This  suggests  that,  in  hindsight,  the  leader  may 
have  been  the  most  dominant  factor  in  the  success  of  a  mission,  whereas  during  the  mission  the 
team  seems  to  be  more  important. 

•  Fourth,  in  Joint  Caribbean  Lion  task-focused  behaviours  had  a  significant  contribution  to  task 
outcomes,  but  not  to  team  outcomes,  whereas  team-focused  behaviours  only  had  a  significant 
contribution  to  team  outcomes  and  not  to  task  outcomes.  It  means  that  the  distinction  between  the 
concepts  task  and  team  may  have  been  better  preserved  in  the  responses  to  an  ongoing  mission. 

Finally,  we  tested  if  the  responses  to  the  survey  were  less  lenient  than  the  responses  to  the  Joint  Caribbean 
Lion  Survey  (see  Table  5-3). 


Table  5-3:  Comparison  of  the  Mean  Values  of  the  Aspects/Components 
that  Were  Measured  in  the  Survey  and  Joint  Caribbean  Lion. 


Survey 

Mean 

Survey 

SD 

JCL 

Mean 

JCL 

SD 

t 

P 

Mission  Framework 

3.14 

0.72 

2.96 

0.57 

2.73 

.003 

Task 

3.05 

0.69 

3.36 

0.63 

-4.87 

.000 

Organisation 

3.49 

0.73 

3.22 

0.58 

4.13 

.000 

Leader 

3.73 

0.96 

3.40 

0.72 

3.13 

.001 

Team  Members 

3.63 

0.72 

3.29 

0.67 

4.54 

.000 

Team 

3.57 

0.59 

2.93 

0.69 

8.50 

.000 

Task-Focused  Behaviours 

3.67 

0.67 

3.00 

0.58 

10.08 

.000 

Team-Focused  Behaviours 

3.57 

0.69 

2.97 

0.62 

9.18 

.000 

Task  Outcomes 

4.07 

0.72 

3.36 

0.56 

10.65 

.000 

Team  Outcomes 

3.79 

0.85 

3.15 

0.73 

8.15 

.000 

Table  5-3  shows  that  the  values  of  the  survey  were  significantly  and  substantially  higher  than  the  values  of 
Joint  Caribbean  Lion.  The  differences  were  largest  for  the  task-  and  team-focused  behaviours  and  outcomes. 
The  only  exception  is  task  characteristics.  These  differences  in  mean  values  suggest  that  respondents  were 
far  more  positive  in  hindsight  than  they  were  during  a  real  exercise. 

In  sum,  we  can  conclude  that  the  responses  in  the  survey  showed  effects  that  could  be  attributed  to  hindsight 
bias.  The  inter-item  correlations  for  most  aspects  and  components  in  the  survey  were  higher  than  in  the  Joint 
Caribbean  Lion  data.  The  higher  correlations  in  the  survey  data  compared  with  those  of  Joint  Caribbean  Lion 
suggest  that  when  the  items  are  filled  out  during  an  exercise  or  a  mission,  respondents  are  better  able  to 
distinguish  between  the  several  items  than  when  respondents  fill  out  the  survey  (long)  after  the  exercise  or 
mission. 

5.3  FURTHER  DEVELOPMENT  OF  MODEL  AND  INSTRUMENT:  CTEF  2.0 

The  analyses  in  Chapter  4  showed  that  the  length  of  the  instrument  could  be  reduced  because  the  inter¬ 
item  correlations  were  high  in  the  survey  and  in  Joint  Caribbean  Lion.  In  this  section  we  describe  our 
decisions  working  toward  the  next  version  of  the  instrument. 


5-6 


RTO-TR-HFM-127 


DISCUSSION 


5.3.1  Remove  Impact  Scale 

The  first  reduction  of  the  questionnaire  required  the  removal  of  either  the  magnitude  scale  or  the  impact 
scale.  On  the  basis  of  the  high  inter-correlations  between  both  scales  we  decided  that  it  is  not  necessary  to 
use  both  scales.  We  decided  to  keep  the  magnitude  scale  because  this  scale  refers  to  the  extent  to  which  an 
item  is  present  in  the  team  and  because  it  appears  throughout  the  questionnaire;  that  is,  there  is  no  impact 
scale  for  the  outcome  measures. 

5.3.2  Remodel  the  Components  Mission  Framework  and  Task 

The  regression  analyses  showed  that  the  aspects  mission  framework  and  task  consistently  did  not  show 
significant  effects  on  task-  and  team-focused  behaviours  and  outcomes.  Further  exploration  of  the  effects 
of  the  items  within  both  aspects  showed  that  the  features  within  each  of  these  aspects  cancelled  each  other 
out;  i.e.,  they  had  contrary  effects  on  the  task-  and  team-focused  behaviours  and  outcomes. 

The  correlations  in  Table  5-4  show  a  number  of  interesting  aspects.  First,  some  of  the  items  of  mission 
framework  were  highly  correlated  with  the  items  of  task  characteristics:  operational  uncertainty  was 
highly  correlated  with  goal  ambiguity  and  goal  instability;  operational  stress  was  also  highly  correlated  - 
although  somewhat  lower  -  with  complexity  and  workload.  Second,  items  within  both  mission  framework 
and  task  characteristics  had  conflicting  correlation  structures  with  task-  and  team-focused  behaviours  and 
task  and  team  outcomes.  Whereas  situational  uncertainty  correlated  negatively  with  the  behaviour  and 
outcome  items,  mission  stakes  correlated  positively  with  them.  For  task  characteristics  the  correlations  of 
workload  and  complexity,  on  the  one  hand,  and  goal  ambiguity  and  goal  instability,  on  the  other  hand, 
contradicted  each  other.  These  contradicting  correlations  could  be  the  reason  why  the  overall  concepts 
mission  framework  and  task  did  not  have  an  effect  on  the  task-  and  team-focused  behaviours  and  task  and 
team  outcomes. 


Table  5-4:  Correlation  Coefficients  between  the  Features  of  Mission  Framework  and  Task, 
and  the  Aspects  of  Task  and  Team-Focused  Behaviours,  and  Task  and  Team  Outcomes. 


1 

2  3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

Mission  Framework 

Situational  Uncertainty 

1 

.23  .29 

.13 

.10 

.10 

J>0 

.37 

-.25 

-.22 

-.25 

-.27 

Operational  Stress 

2 

.34 

.35 

.36 

,50 

.19 

.20 

.05 

.02 

-.00 

-.06 

Constraints 

3 

.27 

.25 

.20 

.20 

.21 

-.02 

-.04 

-.01 

-.13 

Mission  Stakes 

4 

.29 

.39 

.01 

.08 

.17 

.17 

.13 

.13 

Task  Characteristics 

Complexity 

5 

A2 

.11 

.14 

.12 

.12 

.06 

.08 

Workload 

6 

-.00 

.09 

.19 

.22 

.20 

.14 

Goal  Ambiguity 

7 

.61 

-.38 

-.33 

-.35 

-.38 

Goal  Instability 

8 

-.27 

-.23 

-.25 

-.28 

Task-Focused  Behaviours 

9 

.80  .71 

.75 

Team-Focused  Behaviours 

10 

.65 

.89 

Task  Outcomes 

11 

.64 

Team  Outcomes 

12 
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However,  the  three  features  situational  uncertainty  (mission  framework),  and  goal  ambiguity,  and  goal 
instability  (task)  all  had  the  same  consistent  effects.  They  showed  strong  negative  relationships  with  both  the 
task-  and  team-focused  behaviours  as  well  as  with  the  task  and  team  outcomes.  A  comparable  merge  of  the 
items  workload  and  complexity  can  be  suggested.  These  items  also  have  comparable  correlation  structures. 

On  the  basis  of  this  analysis  we  suggest  to  redefine  the  existing  components  mission  framework  and  task  into 
mission  context  and  task  characteristics.  Mission  context  is  oriented  on  the  environment  and  the  mission 
goals.  The  constituting  aspects  are  situational  uncertainty  and  goal  instability.  Goal  ambiguity  was  not  used 
because  it  had  a  relatively  high  correlation  with  Situation  uncertainty,  becoming  a  too  similar  construct. 
Task  characteristics  are  focused  on  the  work  itself.  Its  aspects  are  now  workload  and  complexity. 
The  remaining  items  for  CTEF  2.0  are: 

•  Situational  uncertainty;  and 

•  Goal  instability. 

5.3.3  Organisation 

The  component  organisation  impacted  the  mediating  and  dependent  variables  in  the  model.  Therefore, 
the  aspect  should  be  maintained  as  such.  A  further  analysis  showed  that  the  item  freedom  of  action 
(autonomy)  correlated  lower  with  the  other  items  than  the  other  three  items  that  comprised  this 
component.  Therefore,  we  chose  not  to  include  freedom  of  action  in  the  CTEF  2.0.  The  remaining  items 
for  CTEF  2.0  are: 

•  Clarity  of  command  structure; 

•  Support  provided  by  the  organisation;  and 

•  Match  between  the  team  mission  and  organisational  goals. 

5.3.4  Team  Leader 

In  the  survey,  the  component  team  leader  had  a  large  effect  on  task-  and  team-focused  behaviours  and 
outcomes.  However,  the  aspects  team  leader  skill  and  team  leader  knowledge  correlated  very  highly, 
suggesting  that  respondents  were  not  able  to  assess  the  subtle  differences  between  both.  Therefore, 
we  decided  to  combine  both  aspects  into  the  item  team  leader  competence.  The  items  for  CTEF  2.0  are  now: 

•  Team  leader  competencies;  and 

•  Match  between  leader’s  personal  goals  and  organisational  goals. 

5.3.5  Team  Members 

Just  as  in  the  component  team  leader,  the  aspects  team  members’  skill  and  team  members’  knowledge 
correlated  very  highly,  suggesting  that  respondents  were  not  able  to  assess  the  subtle  differences  between 
both.  Therefore,  we  also  decided  to  combine  both  features  in  the  item  team  members’  competence. 
The  items  for  CTEF2.0  are  now: 

•  Team  members’  competence;  and 

•  Match  between  members’  personal  goals  and  organisational  goals. 

5.3.6  Team 

The  aspect  team  had  high  beta  values  in  the  regression  equations.  Therefore,  this  aspect  is  considered 
important  in  the  model.  A  further  analysis  shows  that  the  inter-item  correlations  of  team  size  and  face-to-face 
interaction  with  the  other  items  were  considerably  lower  than  the  inter-item  correlations  of  the  other  items. 
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Furthermore,  the  item  team  size  is  not  specific  about  the  quality  of  the  team.  Therefore,  we  decided  to  leave 
this  feature  team  size  out.  The  face-to-face  interaction  item  was  skewed:  only  five  percent  of  the  respondents 
were  actually  working  totally  distributed.  Therefore,  we  did  not  include  face-to-face  interaction  in  the  CTEF 
2.0.  It  is  important  to  consider  that  mix  of  people  and  team  structure  only  can  be  scored  in  a  positive  or 
negative  way  if  an  anchor  such  as  “appropriateness”  is  used.  The  items  for  CTEF  2.0  are: 

•  Mix  of  people; 

•  Team  structure; 

•  Team  maturity;  and 

•  Match  between  team  goals  and  organisational  goals. 

5.3.7  Task-Focused  Behaviours 

The  inter- item  correlations  of  the  19  items  were  high.  Moreover,  at  feature  level  (Level  4)  within  each 
aspect  they  were  higher  as  compared  to  the  inter-item  correlations  between  the  different  aspects  (Level  3). 
Therefore,  we  decided  to  compress  the  number  of  items  and  not  use  the  Level  4  (the  features)  in  CTEF 
2.0.  The  items  that  are  now  in  CTEF  2.0  are: 

•  Managing  information; 

•  Decision  making; 

•  Planning; 

•  Executing  plans;  and 

•  Interacting  with  other  command  teams. 

5.3.8  Team-Focused  Behaviours 

The  inter-item  correlations  between  the  1 8  items  that  make  up  the  team-focused  behaviours  were  also  high. 
Therefore,  in  general  we  decided  to  compress  the  number  of  items  and  use  the  aspects  level  (Level  3). 
However,  the  first  two  aspects  (providing  and  maintaining  vision  and  maintaining  common  intent)  correlated 
very  highly.  Therefore,  we  combined  them  into  ‘providing  vision  and  intent’.  Another  change  was  suggested 
for  the  aspect  “adapting  to  changes”.  Based  on  what  we  really  want  to  measure  and  what  is  not  already  in 
other  aspects,  we  concluded  that  the  features  monitoring  team  members’  performance  and  providing  back-up 
are  in  fact  more  central  to  what  we  are  trying  to  assess.  Also  a  small  change  in  the  wording  of  monitoring 
team  members’  behaviours  was  introduced  in  order  to  convey  that  we  are  assessing  social  aspects.  Hence, 
the  aspect  “adapting  to  changes”  was  changed  into  “monitoring  team  members’  behaviours”  and  “providing 
back-up”.  Also,  we  decided  to  rephrase  the  aspect  “maintaining  team  synergy”  into  “maintaining  team 
cohesion”,  because  the  item  was  hard  to  understand  for  the  respondents.  In  conclusion,  the  aspects  that  we 
measure  in  CTEF  2.0  are: 

•  Providing  vision  and  intent; 

•  Collaborating  between  team  members; 

•  Motivating; 

•  Monitoring  team  members’  behaviour; 

•  Providing  back-up;  and 

•  Maintaining  team  cohesion. 
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5.3.9  Task  Outcomes 

The  inter-item  correlations  between  the  three  items  were  high,  especially  between  the  items  meeting  goals 
of  the  commander  and  meeting  expectations  of  other  partners.  We  decided  to  merge  these  two  items 
because  of  this  high  correlation  and  because  of  the  argument  that  for  the  military  the  expectations  of  other 
partners  are  generally  covered  by  the  goals  of  the  commander.  The  remaining  two  items  that  we  measure 
in  CTEF2.0  are  then: 

•  Meeting  the  goals  of  the  commander  and  higher  echelons;  and 

•  Staying  within  limits  and  intentions  of  the  mission. 

5.3.10  Team  Outcomes 

The  inter-item  correlations  were  very  high.  Therefore,  we  decided  to  reduce  the  number  of  items  from  six  to 
two  items.  The  remaining  items  should  reflect  an  internal  dimension  and  an  external  dimension.  We  decided 
to  keep: 

•  Trust  between  team  members  (internal);  and 

•  Collective  confidence  in  achieving  goals  (external). 

5.3.11  Feedback 

The  inter-item  correlations  were  extremely  high,  suggesting  that  the  respondents  could  not  differentiate 
between  the  items  or,  if  they  use  one  form  of  feedback,  they  use  the  other  forms  as  well.  Also,  there  was  a 
concern  in  the  research  group  that  the  wording  was  not  clear  enough.  Given  that  the  processes  are  the  core  of 
the  model,  we  decided  to  select  one  item  for  task  processes  and  one  item  for  team  processes.  The  items  are: 

•  Did  your  team  take  measures  to  improve  task  processes  when  needed 

(managing  information,  decision  making,  planning,  executing  plans,  interacting  with  other  command 
teams); 

•  Did  your  team  take  measures  to  improve  team  processes  when  needed 

(providing  vision  and  intent,  collaboration  among  team  members,  motivating,  monitoring  team 
members’  behaviour,  providing  back-up,  maintaining  team  cohesion). 

5.3.12  Conclusion 

The  above  reductions  resulted  in  a  CTEF  2.0  instrument  of  32  items  that  ask  for  the  magnitude  or  level  or 
appropriateness  of  certain  aspects.  The  answer  categories  that  are  used  for  the  items  are  the  same  that  were 
used  for  the  administrations  of  the  survey. 


5.4  USE  OF  THE  MODEL  AND  THE  INSTRUMENT 

The  primary  use  of  the  instrument  is  to  discuss  the  results  by  the  commander  and  his  or  her  team. 
The  commander  needs  to  specify  which  collectives  are  the  ‘teams’  to  be  considered.  For  staffs,  it  is 
suggested  to  specify  a  team  comprising  the  commander,  the  chief  of  staff,  and  the  section  heads.  The  other 
teams  in  a  staff  can  be  sections  or  sub-sections  to  the  extent  that  they  work  closely  together  to  achieve 
specific  common  goals. 

The  instrument  can  be  used  in  different  ways  (see  Annex  F).  For  example,  in  exercise  Joint  Caribbean  Fion, 
we  grouped  items  in  order  to  arrive  at  global  ‘negative’  and  ‘positive’  items.  We  used  a  simple  criterion 
whereby  the  item  should  be  flagged  for  discussion  if  more  than  40%  of  the  team  agreed  that  it  was  a 
significant  issue.  By  doing  so,  the  staff  briefing  can  focus  exclusively  on  the  critical  issuers.  During  the 
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briefing,  the  team  should  try  to  uncover  and  understand  why  these  aspects  or  features  have  these  scores  and 
what  could  be  done  to  either  improve  the  aspect  or  feature  or  mitigate  its  negative  effects  on  team 
effectiveness.  If  the  instrument  is  used  repeatedly,  then  development  or  improvement  can  be  tracked. 

The  CTEF  instrument  can  be  used  at  any  one  of  several  stages  of  a  team’s  development.  Shortly  after  a 
team  is  formed,  application  of  the  instrument  could  be  beneficial  in  alerting  and  informing  the  team  leader 
and  members  about  aspects  of  team  performance  which  might  support  (or  be  detrimental  to)  the  team’s 
task.  This  would  allow  ‘self  regulation’  of  team  context,  people  and  process  aspects,  at  least  to  the  extent 
that  these  are  under  the  control  of  the  team.  As  a  team  matures,  it  might  be  useful  to  do  a  status  check  to 
support  internal  or  external  adjustment  for  the  sake  of  improved  effectiveness.  A  readiness  check  on  a 
‘mature’  team  may  provide  a  senior  commander  with  understanding  of  likely  team  effectiveness  when 
faced  with  an  upcoming  challenge.  For  example,  if  a  provisional  task  force  headquarters  has  been  formed 
for  particular  contingencies,  it  might  be  appropriate  to  assess  the  command  team’s  readiness  in  parallel 
with  assessments  of  subordinate  units’  readiness  for  deployment.  A  status  check  for  an  active  team  in  the 
midst  of  an  ongoing  task  might  provide  useful  feedback  on  ways  to  improve  team  effectiveness. 
Utilisation  of  the  instrument  following  completion  of  a  major  task  could  identify  lessons  helpful  in  the 
formation  of  a  similar  team  in  the  future.  In  general,  the  instrument  may  be  used  either  to  assess  current 
status  or  readiness  at  virtually  any  stage  in  the  life  cycle  of  a  team. 

The  instrument  can  be  used  in  different  roles:  by  observers,  leaders  and  team  members.  In  general  observers 
score  most  severely,  and  leaders  score  most  positively.  These  differences  are  “natural”  and  provide  a  solid 
basis  for  discussion:  not  who  is  right,  but  what  are  the  issues  behind  the  scores,  items,  concepts  of  the 
instrument  and  model.  In  summary,  the  emphasis  of  this  tool  is  not  to  provide  a  definitive  or  final 
judgment  on  the  team.  It  is  meant  rather  to  discover  ways  to  improve  the  team  and  to  sensitise  the  team  to 
important  issues. 
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We  developed  a  comprehensive  model  for  Command  Team  Effectiveness  (CTEF)  and  a  corresponding 
assessment  instrument  based  on  an  extensive  literature  review  on  team  effectiveness  and  analysis  of 
operational  command  teams,  meant  to  support  operational  commanders  and  teams.  This  development  started 
out  with  large  set  of  elements  that  was  judged  to  be  critically  relevant  for  operational  performance.  Through 
applications  in  practice  and  analysis  of  experiences  of  718  surveyed  military  officers.  This  process  allowed 
us  to  reduce  the  number  of  elements  in  the  model  -  from  75  to  32  items  -  on  the  basis  of  inter-item 
correlations,  advanced  modelling  techniques  and  operational  relevance.  This  resulted  in  an  improved  version 
of  the  command  team  effectiveness  instrument,  the  CTEF  2.0. 

It  is  important  to  realise  that  the  basic  structure  of  the  CTEF  model  and  its  constituting  components  were 
not  changed  substantially;  rather,  to  assess  these  components  fewer  detailed  aspects  and  features  are 
needed  according  to  our  extensive  analysis.  CTEF  2.0  is  now  the  basis  for  further  application  and  data 
gathering  in  operational,  exercise,  and  training  contexts. 

Commanders  and  leaders  always  have  to  decide  themselves  what  level  of  detail  is  needed  to  build  their 
teams,  and  they  can  add  more  detailed  items  from  the  model  in  order  to  tap  deeper  in  the  performance  of 
their  team. 

It  is  recommended  that  commanders  learn  to  work  with  the  effectiveness  concepts  proposed  by  the  CTEF 
model  and  apply  it  regularly,  possibly  selecting  those  elements  they  want  to  work  on  to  improve. 
For  instance,  the  NATO  schools  could  use  it  in  their  training  courses  and  academies  and  staff  colleges  in 
their  leadership  courses  as  well. 

We  recommend  that  a  simple  support  system  is  developed  for  easy  data  collecting,  processing  data, 
and  reporting. 

Finally,  we  recommend  that  follow-up  development  of  CTEF  be  initiated,  specifically  directed  to  more 
complex  organisations,  such  as  international  staffs  and  multi-team  organisations  in  a  comprehensive 
context  of  military-civil  alliances. 
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Annex  A  -  THE  CTEF  INSTRUMENT  BASED  ON  CTEF  2005 
MODEL  AS  ADMINISTRATED  IN  JOINT  CARIBBEAN  LION 


Quality /Magnitude 

Please  evaluate  the  quality  or  magnitude  of 
the  aspect  being  rated  using  the  following 

scale: 

Very 

Moderate  Vei^ 

Low 

High 

i 

2  3  4  5 

Impact  on  Team  Effectiveness 

Please  evaluate  the  direction  and  strength  of  impact  on 
overall  team  effectiveness  due  to  the  present  level  of  an 
item’s  quality  or  magnitude: 

Very  No  Very 

Negative  Impact  Positive 

-2  -1  0  +1  +2 


Quality/Magnitude  Impact 


MISSION  FRAMEWORK 

Very 

Very 

Very 

Very 

Low 

High 

Negative 

Positive 

Level  of  situational  uncertainty 

The  lack  of  information  and  lack  of  understanding  of  the  operation 

l 

2 

3 

4 

5 

-2  -1 

0 

+1 

+2 

Level  of  operational  stress 

The  state  of  mental  or  emotional  strain  during  the  operation 

l 

2 

3 

4 

5 

-2  -1 

0 

+1 

+2 

Amount  of  limitations 

External  factors  that  restrict  the  range  of  the  team ’s  actions  or  independence 

i 

2 

3 

4 

5 

-2  -1 

0 

+1 

+2 

Level  of  risk  of  damaging  interests 

The  implications  of  the  success  or  failure  of  the  mission 

i 

2 

3 

4 

5 

-2  -1 

0 

+1 

+2 

TASK 

Very  Very 

Low  High 

Very  Very 

Negative  Positive 

Level  of  task  complexity 

Tasks  that  have  many  related  or  difficult  subtasks 

Level  of  workload 

Refers  to  the  mental  and  physical  demands  of  the  task 

Level  of  goal  uncertainty 

The  degree  to  which  the  mission  goals,  objectives,  and  priorities  are  unknown, 
vague,  or  unclear 

Level  of  goal  instability 

The  degree  to  which  goals  change  significantly  over  time 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

-2  -1  0  +1  +2 

-2  -1  0  +1  +2 

-2  -1  0  +1  +2 

-2  -1  0  +1  +2 

ORGANISATION 

Very  Very 

Very  Very 

Low  High 

Negative  Positive 

Match  between  the  team’s  mission  and  organisational  goals 

The  degree  to  which  team  s  mission  is  similar  to  the  organisation  s  goals 

Clarity  of  command  structure 

Understanding  the  command  structure  and  ability  to  prioritize  when  given  orders 
from  multiple  chains  of  command  (e.g,  staff  and  line  versus  matrix  structure) 

Level  of  autonomy 

Freedom  of  action  allowed  by  the  organisation 

Level  of  organisational  support 

Degree  to  which  the  oiganisation  assists  and  sustains  the  team 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

-2  -1  0  +1  +2 

-2  -1  0  +1  +2 

-2  -1  0  +1  +2 

-2  -1  0  +1  +2 

TEAM  LEADER 

Very 

Very 

Very 

Very 

Low 

High 

Negative 

Positive 

Level  of  leader  skills 

Essential  leader  capabilities  as  they  relate  to  the  military  task  at  hand 

i 

2 

3 

4 

5 

-2  -1 

0 

+1 

+2 

Level  of  leader  knowledge 

Essential  knowledge,  wisdom  and  experience  of  the  leader 

i 

2 

3 

4 

5 

-2  -1 

0 

+1 

+2 

Consistency  between  the  leader’s  goals  and  the  organisational  goals 

The  degree  to  which  the  leader ’s  goals  are  similar  to  the  organisation ’s  goals 

i 

2 

3 

4 

5 

-2  -1 

0 

+1 

+2 
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CTEF  2005  MODEL  AS  ADMINISTRATED  IN  JOINT  CARIBBEAN  LION 


ORGANIZATION 


Quality/Magnitude  Impact 


TEAM  MEMBERS 

Very 

Low 

Very 

High 

Very 

Negative 

Very 

Positive 

Level  of  team  member  skills 

Abilities  held  by  individual  team  members  which  enable  them  to  complete  their 
tasks  within  a  team  setting 

1 

2 

3 

4 

5 

-2  -1 

0 

+1 

+2 

Level  of  team  member  knowledge 

Knowledge,  wisdom  and  experience  of  the  team  members 

i 

2 

3 

4 

5 

-2  -1 

0 

+1 

+2 

Consistency  of  team  member’s  goals  to  organisational  goals 

The  degree  to  which  team  members '  goals  are  similar  to  the  organisation ’s  goals 

i 

2 

3 

4 

5 

-2  -1 

0 

+  1 

+2 

TEAM 

Very  Very 

Low  High 

Very  Very 

Negative  Positive 

Appropriateness  of  team  composition 

Degree  to  which  the  mix  of  team  member’s  skills  and  personalities  match  the  needs 
of  the  mission  requirements 

Appropriateness  of  team  size 

Degree  to  which  the  team  is  sufficiently  staffed  to  accomplish  the  task 

Appropriateness  of  team  architecture 

Distribution  and  relation  of  subtasks  and  roles  between  team  members 

Level  of  team  maturity 

Degree  to  which  team  members  have  worked  together  and  developed  as  intact  team 

Consistency  of  team  goals  to  organisational  goals 

The  degree  to  which  the  team  goals  are  similar  to  the  organisational  goals 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

-2  -1  0  +1  +2 

-2  -1  0  +1  +2 

-2  -1  0  +1  +2 

-2  -1  0  +1  +2 

-2  -1  0  +1  +2 

TASK  FOCUSED  BEHAVIOURS 

Very  Very 

Low  High 

Very  Very 

Negative  Positive 

Quality  of  managing  information 

How  well  the  team  obtains,  uses,  coordinates  and  exchanges  information  in  order  to 
understand  the  situation 

1  2  3  4  5 

-2  -1  0  +1  +2 

Quality  of  obtaining  information 

i.e.,  actively  searching  to  compensate  for  the  lack  of  information 

Quality  of  processing  information 

e.g.,  using,  integrating  information;  maintain  focus 

Quality  of  exchanging  information 

e.g.,  timeliness,  clarity,  brevity,  correctness,  completeness  ofcomms. 

Quality  of  assessing  the  situation 

e.g.,  noticing,  recognising,  anticipating  events 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

-2  -1  0  +1  +2 

-2  -1  0  +1  +2 

-2  -1  0  +1  +2 

-2  -1  0  +1  +2 

Quality  of  making  decisions 

How  well  the  team  creates  multiple  solutions,  chooses  among  solutions,  and 
implements  the  chosen  solution 

1  2  3  4  5 

-2  -1  0  +1  +2 

Quality  of  defining  the  problem  space 

e.g.,  sampling  environmental  cues  and  team  members’  expertise 

Quality  of  managing  time  for  decision-making 

e.g.,  adapting  to  available  time 

Quality  of  evaluating  options  and  results 

e.g.,  process  of  continually  thinking,  acting,  evaluating  outcomes 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

-2  -1  0  +1  +2 

-2  -1  0  +1  +2 

-2  -1  0  +1  +2 

Quality  of  planning 

How  well  the  team  outlines  actions  required  fora  task  and  identifies  &  schedule 
available  resources 

1  2  3  4  5 

-2  -1  0  +1  +2 

Quality  of  anticipating 

e.g.,  identifying  future  tasks  and  possible  outcomes 

Quality  of  scheduling 

e.g.,  personnel  and  resources  in  time 

Quality  of  specifying  resources  needed 

e.g.  personnel,  time,  tools 

Quality  of  defining  strategies 

defining  an  approach  to  accomplish  the  mission 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

-2  -1  0  +1  +2 

-2  -1  0  +1  +2 

-2  -1  0  +1  +2 

-2  -1  0  +1  +2 

Quality  of  directing  and  controlling 

Processes  that  occur  between  planning  and  attaining  a  goal 

1  2  3  4  5 

-2  -1  0  +1  +2 

Quality  of  organising 

to  implement  the  plan  specifying  resources,  procedures,  roles 

Quality  of  managing 

managing  time,  schedules,  and  expectations  in  executing  decisions 

Quality  of  monitoring  progress 

assessing  advancement  toward  milestones,  goals,  and  objectives 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

-2  -1  0  +1  +2 

-2  -1  0  +1  +2 

-2  -1  0  +1  +2 

Quality  of  interactions  with  other  command  teams 

Developing  and  maintaining  communication  with  other  command  teams 

1  2  3  4  5 

-2  -1  0  +1  +2 
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CTEF  2005  MODEL  AS  ADMINISTRATED  IN  JOINT  CARIBBEAN  LION 


Quality/Magnitude  Impact 


TEAM-FOCUSED  BEHAVIOURS 

Very 

Very 

Very 

Very 

Low 

High 

Negative 

Positive 

Quality  of  providing  and  maintaining  vision 

i 

0 

+1 

+2 

Communicating  the  team’s  direction  and  purpose 

Quality  of  maintaining  common  intent 

i 

2 

3 

4 

5 

-2  -1 

0 

+1 

+2 

Creating  a  shared  sense  of  goals,  objectives,  and  action 

Quality  of  interacting  within  the  team 

i 

-2  -1 

0 

+1 

+2 

Communication  and  coordination  within  the  team  and  providing  feedback 

Quality  of  communicating 

e.g.  openness,  style,  expressing  feelings  and  thoughts 

i 

2 

3 

4 

5 

-2  -1 

0 

+1 

+2 

Quality  of  coordinating 

Combining/acting  in  a  harmonious,  supportive  manner  to  achieve  goals 

i 

2 

3 

4 

5 

-2  -1 

0 

+1 

+2 

Quality  of  providing  feedback 

Offering  advice  to  one  other  about  how  to  improve  performance 

i 

2 

3 

4 

5 

-2  -1 

0 

+1 

+2 

Quality  of  Motivating 

i 

2 

3 

4 

5 

-2  -1 

0 

+1 

+2 

Influencing  the  amount  of  effort  exerted  toward  achieving  the  team 's  goals 

Quality  of  Motivating  Externally 

i 

1 

3 

4 

5 

-?  -1 

0 

+1 

+2 

Using  tangible /intangible  incentives  to  reward  good  performance 

Quality  of  Motivating  Internally 

i 

0 

+1 

+2 

Inspiring  its  members  to  develop  internal  interest  in  work 

j 

Z  1 

Quality  of  Adapting 

i 

2 

3 

4 

5 

-2  -1 

0 

+1 

+2 

Monitoring  task  and  team  status  and  adjusting  strategies  when  needed 

Quality  of  monitoring 

Observing  and  assessing  own  and  each  other ’s performance 

i 

2 

3 

4 

5 

-2  -1 

0 

+1 

+2 

Quality  of  correcting 

Offering  feedback  or  guidance  to  improve  team  mates'  performance 

i 

2 

3 

4 

5 

-2  -1 

0 

+1 

+2 

Quality  of  backing-up 

Supporting  another  team  member  when  s/he  is  busy 

i 

2 

3 

4 

5 

-2  -1 

0 

+1 

+2 

Quality  of  providing  team  maintenance 

i 

2 

3 

4 

5 

-2  -1 

0 

+1 

+2 

Team  building  activities  that  keep  the  team  together 

Quality  of  providing  social  support  and  integration 

e.g.,  respect  and  companionship 

i 

2 

3 

4 

5 

-2  -1 

0 

+1 

+2 

Quality  of  regulating  emotions  (e.g.,  composure,  morale) 

Maintaining  emotional  balance  among  team  members,  considering 
emotional  display  norms 

i 

2 

3 

4 

5 

-2  -1 

0 

+1 

+2 

Quality  of  developing  and  maintaining  cohesion 

i 

0 

+1 

+2 

Promoting  unity,  solidarity,  or  esprit  de  corps  among  themselves 

J 

Quality  of  managing  conflict 

i 

2 

3 

-2  -1 

0 

+1 

+2 

Conflict,  argument  or  friction  is  eliminated  or  reduced 

TASK  OUTCOMES 

Very 

Low 

Very 

High 

Very 

Negative 

Very 

Positive 

Degree  to  which  the  primary  stakeholder’s  criteria  are  met  (stakeholder’s  can 
include  superior  commanders) 

i 

2 

3 

4 

5 

-2  -1 

0 

+  1 

+2 

e.g.  timeliness,  completeness 

Degree  to  which  other  stakeholders’  expectations  are  met  (stakeholder’s  can 
include  superior  commanders) 

i 

2 

3 

4 

5 

-2  -1 

0 

+  1 

+2 

Achievement  of  expectancies  of  parties  involved 

Degree  to  which  the  team  stays  within  the  limits/  intentions 

Obeying  rules  and  limits  set 

i 

2 

3 

4 

5 

-2  -1 

0 

+  1 

+2 

TEAM  OUTCOMES 


Very 

Low 


Level  of  mutual  trust  j 

Team  members  believe  in  each  other’s  competence,  loyalty,  and  dedication 
Level  of  morale  ^ 

Degree  to  which  team  members  have  a  positive  attitude  toward  the  team 

Level  of  cohesion 

The  team  members  act  as  a  team  instead  of  as  individuals,  they  feel  attracted  1 

towards  the  team 

Level  of  collective  confidence  in  achieving  the  goal  j 

Team  members  have  a  strong  belief  in  the  effectiveness  of  the  team 

Level  of  shared  vision 

In  general,  team  members  view  the  task  that  they  have  to  accomplish  in  the  same  1 

way 

Level  of  mutual  respect  ^ 

Despite  differences,  the  team  members  try  to  understand  each  other 


2 

2 

2 

2 

2 

2 


Very 

High 

Very  Very 

Negative  Positive 

3  4  5 

-2  -1  0  +1  +2 

3  4  5 

-2  -1  0  +1  +2 

3  4  5 

-2  -1  0  +1  +2 

3  4  5 

-2  -1  0  +1  +2 

3  4  5 

-2  -1  0  +1  +2 

3  4  5 

-2  -1  0  +1  +2 
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Annex  B  -  THE  CTEF  INSTRUMENT  BASED  ON  CTEF 
2005  MODEL  AS  EMPLOYED  IN  THE  NATO  HFM 
TG-127  MULTI-NATIONAL  SURVEY  STUDY 


Command  Team  Effectiveness  Instrument 

A  tool  for  assessment  &  improvement 

(NATO  RTO  HFM  TG127) 


Version  19  Nov  2007 


Foreword  and  recommendation 

‘Effective  teamwofin  command  teams  is  a  criticaC  mission  success  factor.  ‘Therefore,  commanders  need 
to  gain  and  maintain  insight  into  the  effectiveness  of  their  teams.  They  need  to  understand  how  to 
foster  and  devefop  the  teamworhjwithin  the  command  staff  as  wed  as  with  their  suh -commanders.  JAs 
a  Commander  first  of  Submarine  operations,  and  Cater  of  Jimp  hi6ious  operations,  I  highCy  vaCuedthe 
importance  of  the  collective  expertise,  commitment,  and  motivation  of  my  su6-  commanders  and  staffs. 

TChen  I  came  across  the  JfJTTO  Command  Team  Effectiveness  modeC  (CTTEE)  I  immediateCy 
recognized  its  vaCue  and  decided  to  appCy  it  in  a  five  exercise  (‘Joint  Cari66ean  Lion’  CMay  2006). 
During  this  five  exercise,  CTTEE  data  was  gathered  at  three  different  points  in  time.  Each  time  the 
same  night  the  data  was  sent  over  to  the  CTTEE  deveCopment  team,  who,  working  as  a  reaC distributed 
team  on  two  continents,  anaCysed  the  data  and  returned  the  resuCts  the  day  after.  In  this  way  I  had 
timeCy,  structured  and  wed  founded  information  to  prepare  de6riefs  and  discuss  with  the  staff  the 
(reCative)  state  of  the  conditions,  the  processes  and  the  outcomes,  ena6Cing  a  comprehensive  approach 
to  continuous  improvement  of  team  effectiveness. 

I  can fuCCy  recommend this  survey,  as  an  effort  to  further  deveCop  and  vaCidate  the  CTEE  modeC.  This 
wiff head  to  effective  support  for  operationaC  commanders  in  the  essence  of  his  orherjo6  -  guiding  and 
fostering  effective  command  teams. 


Commodore  Pieter  Bindt. 
Commander  Netherlands  Maritime  Force. 

Den  Helder,  August  15,  2007 
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ORGANIZATION 


Command  Team  Effectiveness  Survey 


Questions  on  the  following  pages  will  ask  you  to  reflect  on  the  performance  and  effectiveness  of  a 
command  team  you  have  been  a  part  of,  as  leader  or  member. 

Please  consider  one  command  team  that  you  have  been  part  of,  in  a  recent  deployment  or  other 
operational  context.  Remember  that  this  team  will  be  the  reference  of  ALL  the  upcoming 
questions. 

First,  we  will  ask  you  to  describe  this  command  team.  Then,  we  will  ask  you  to  evaluate  the  team  with  a 
number  of  teamwork  related  factors.  Finally,  we  will  ask  you  for  feedback  on  the  usefulness  of  the 
survey. 

Please  answer  openly  and  honestly.  You  are  responding  anonymously  and  your  answers  will  be  treated 
confidentially.  Your  responses  will  only  be  seen  by  the  researchers  -  not  your  commanders  or  other 
military  personnel.  Individual  responses  will  not  be  released;  all  results  will  be  reported  at  the  group 
level. 
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Please  describe  the  main  tasks  performed  by  this  team: 


How  many  team  members  (including  yourself)  were  on  the  team? _ 

Please  indicate  in  months  how  long  most  people  on  this  team  (including  yourself)  have  worked 
together  as  a  team. 


What  nationalities  (by  citizenship)  were  on  this  team  (check  all  that  apply)? 


0  Albania 

0  Czech  Republic 

0  Ireland 

0  Norway 

0  Sweden 

0  Armenia 

O  Denmark 

0  Italy 

0  Poland 

O  Switzerland 

0  Austria 

O  Estonia 

0  Kazakhstan 

O  Portugal 

0  Tajikistan 

0  Azerbaij  an 

O  Finland 

0  Kyrghyz  Republic 

0  Republic  of 

Macedonia 

0  Turkey 

0  Belarus 

O  France 

0  Latvia 

0  Romania 

O  Turkmenistan 

0  Belgium 

0  Georgia 

0  Lithuania 

0  Russia 

0  Ukraine 

0  Bosnia  & 

Herzegovina 

O  Germany 

0  Luxembourg 

0  Serbia 

0  United 

Kingdom 

0  Bulgaria 

0  Greece 

0  Moldova 

O  Slovakia 

0  United  States 

0  Canada 

O  Hungary 

0  Montenegro 

0  Slovenia 

0  Uzbekistan 

0  Croatia 

0  Iceland 

0  Netherlands 

O  Spain 

What  services  were  on  this  team  (check  all  that  apply)? 


Army 

Navy 

Air  Force 

Marine  Corps 

Other 

0 

0 

0 

0 

o 

If  other,  please  specifiy: 


How  was  the  team  typically  distributed? 


All  located  in  the  same  place 

Several  groups  of  people  in  two  or  more 

Everyone  located  in  different  places 

0 

locations 

0 

0 

How  did  the  team  typically  communicate? 


Face-to-face 

Telephone 

E-mail 

Chat 

Other 

0 

0 

o 

0 

0 

If  other,  please  specifiy: 


How  much  did  the  members  of  your  team  have  to  rely  on  each  other  to  perform  the  assigned 
tasks? 


Not  at  all 

A  little 

Somewhat 

A  lot 

Very  Much 

0 

0 

0 

0 

0 
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INSTRUCTIONS 

For  each  item,  you  are  first  asked: 

1 .  to  assess  the  level  or  degree  (i.e.  'very  low'  to  'very  high')  of  a  teamwork  related  factor  for  your  team,  then 

2.  you  will  be  asked  to  assess  the  impact  (i.e.  positive,  negative,  no  impact)  of  that  level  or  degree  of  the  factor  on  the 
effectiveness  of  your  command  team.  If  you  feel  a  factor  is  not  applicable  or  relevant  to  the  situation  and  team  you  are 
describing,  please  indicate  this  by  selecting  'Not  Applicable'  for  the  factor. 


ORGANIZATION 


Example: 


b.  Operational  stress 

Mental  and  emotional  pressure,  as  well  as  physical 
risks  due  to  the  operation 


Level  or  Degree 

Not  Applicable 

o 

Very  Low 

o 

Low 

o 

Moderate 

o 

High 

o 

Veiy  High 

o 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Very  Negative 

o 

Negative 

o 

No  Impact 

o 

Positive 

o 

Very  Positive 

o 

First:  What  was  the  /eve/  or  degree  of  operational  stress? 

If  you  think  the  most  characteristic  part  of  the  mission  can  be  described  by  low  level  of  stress,  please  indicate  'very  low'  or 
'low',  depending  on  your  estimate.  If  you  think  the  most  characteristic  part  of  the  mission  can  be  described  by  high  level  of 
stress,  please  indicate  'very  high'  or  'high',  also  depending  on  your  estimate.  If  you  think  the  level  or  degree  of  stress  is 
between  high  or  low,  please  indicate  'moderate'.  If  you  feel  that  this  item  is  not  applicable  to  the  team  you  are  describing, 
please  indicate  'Not  Applicable'.  Then  you  continue  to  the  next  item. 

Second:  What  was  the  impact  of  this  level  on  overall  team  effectiveness? 

For  the  question  'Impact  on  overall  team  effectiveness?',  base  your  answer  on  your  response  to  the  level  or  degree.  If,  for 
example,  you  have  answered  that  the  level  of  stress  was  very  high,  you  have  to  consider  the  impact  of  this  very  high  level  of 
stress.  If  you  think  that  this  very  high  level  of  stress  has  positive  consequences  for  team  performance,  please  indicate  'very 
positive'  or  'positive',  depending  on  your  estimate.  If  you  think  that  this  very  high  level  of  stress  has  negative  consequences 
for  team  performance,  please  indicate  'very  negative'  or  'negative',  depending  on  your  estimate.  If  you  think  the  very  high 
level  of  stress  has  no  impact,  please  indicate  this  answer. 
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22.  MISSION  FRAMEWORK 


a.  Situational  uncertainty 

Lack  of  information  about  and  insight  in  the 
different  aspects  of  the  operation 


Level  or  Degree 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  Low 

o 

Very  Negative 

o 

Low 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

High 

o 

Positive 

o 

Very  High 

o 

Very  Positive 

o 

b.  Operational  stress 

Mental  and  emotional  pressure,  as  well  as 
physical  risks  due  to  the  operation 


Level  or  Degree 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  Low 

o 

Very  Negative 

o 

Low 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

High 

o 

Positive 

o 

Very  High 

o 

Very  Positive 

o 

c.  External  factors  constraining  the  team’s 
actions 

Politics,  culture,  environment,  weather, 
media  and  other  factors 


Level  or  Degree 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  Low 

o 

Very  Negative 

o 

Low 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

High 

o 

Positive 

o 

Very  High 

o 

Very  Positive 

o 

d.  Mission  stakes 

Consequences  of  success  or  failure  of  the 
mission  on  (inter)national  interests 


Level  or  Degree 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  Low 

o 

Very  Negative 

o 

Low 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

High 

o 

Positive 

o 

Very  High 

o 

Very  Positive 

o 

23.  TASK 

a.  Task  complexity 

Task  difficulty’,  (inter) dependence  and/or 
interference  with  other  tasks 


Level  or  Degree 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  Low 

o 

Very  Negative 

o 

Low 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

High 

o 

Positive 

o 

Very  High 

o 

Very  Positive 

o 

Note:  Questions  1  to  21  covered  the  demographic  questions.  The  two  questions  labelled  ‘36’  (see  Page  B-14)  resulted  from  a 
typing  error,  which  was  corrected  in  the  data  processing. 
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ORGANIZATION 


b.  Workload 

Mental,  physical  or  emotional  effort  caused 
by  time  pressure,  fatigue,  and  other  factors 


Level  or  Degree 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  Low 

o 

Very  Negative 

o 

Low 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

High 

o 

Positive 

o 

Very  High 

o 

Very  Positive 

o 

c.  Vagueness  of  goals 

Unknown  and  unclear  goals  and  priorities 


Level  or  Degree 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  Low 

o 

Very  Negative 

o 

Low 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

High 

o 

Positive 

o 

Very  High 

o 

Very  Positive 

o 

d.  Instability  of  goals 

Repeated  changes  and  modifications  of  goals 
during  the  mission 


Level  or  Degree 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  Low 

o 

Very  Negative 

o 

Low 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

High 

o 

Positive 

o 

Very  High 

o 

Very  Positive 

o 

24.  ORGANISATION 

a.  Match  between  the  team  mission  and 
organizational  goals 

Team  mission  fits  within  the  goals  of  the 
organisation 


Level  or  Degree 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  Low 

o 

Very  Negative 

o 

Low 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

High 

o 

Positive 

o 

Very  High 

o 

Very  Positive 

o 

b.  Clarity  of  the  command  structure 

Reporting  structure  is  clear  despite  multiple 
chains  of  command 


Level  or  Degree 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  Low 

o 

Very  Negative 

o 

Low 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

High 

o 

Positive 

o 

Very  High 

o 

Very  Positive 

o 
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c.  Freedom  of  action  permitted  by  the 
organisation 

Freedom  to  make  immediate  decisions  in 
unanticipated  situations 


Level  or  Degree 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  Low 

o 

Very  Negative 

o 

Low 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

High 

o 

Positive 

o 

Very  High 

o 

Very  Positive 

o 

d.  Support  provided  by  the  organisation 

Recognition,  resources,  and  technical 
support  provided  by  the  organization 


Level  or  Degree 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  Low 

o 

Very  Negative 

o 

Low 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

High 

o 

Positive 

o 

Very  High 

o 

Very  Positive 

o 

25.  TEAM  LEADER 

a.  Team  leader  skills 

Tactical,  technical,  interpersonal,  and 
reasoning  skills 


Level  or  Degree 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  Low 

o 

Very  Negative 

o 

Low 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

High 

o 

Positive 

o 

Very  High 

o 

Very  Positive 

o 

b.  Team  leader  knowledge 

Knowledge  and  experience  about  tasks, 
team,  and  organization 


Level  or  Degree 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  Low 

o 

Very  Negative 

o 

Low 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

High 

o 

Positive 

o 

Very  High 

o 

Very  Positive 

o 

c.  Match  between  leader’s  personal  goals 
and  organisational  goals 

Leader’s  goals  fit  within  the  goals  of  the 
organisation 


Level  or  Degree 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  Low 

o 

Very  Negative 

o 

Low 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

High 

o 

Positive 

o 

Very  High 

o 

Very  Positive 

o 
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ORGANIZATION 


26.  TEAM  MEMBERS 

a.  Team  members’  skills 

Tactical,  technical,  interpersonal,  and 
reasoning  skills 


Level  or  Degree 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  Low 

o 

Very  Negative 

o 

Low 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

High 

o 

Positive 

o 

Very  High 

o 

Very  Positive 

o 

b.  Team  members’  knowledge 

Knowledge  and  experience  about  tasks, 
team,  and  organization 


Level  or  Degree 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  Low 

o 

Very  Negative 

o 

Low 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

High 

o 

Positive 

o 

Very  High 

o 

Very  Positive 

o 

c.  Match  between  member’s  personal  goals 
and  organisational  goals 

Individual  team  member’s  goals  fit  within 
the  goals  of  the  organisation 


Level  or  Degree 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  Low 

o 

Very  Negative 

o 

Low 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

High 

o 

Positive 

o 

Very  High 

o 

Very  Positive 

o 

27.  TEAM  CHARACTERISTICS 
a.  Mix  of  people 

Mix  of  skills,  experience,  and  personal 
characteristics 


Level  or  Degree 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  Low 

o 

Very  Negative 

o 

Low 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

High 

o 

Positive 

o 

Very  High 

o 

Very  Positive 

o 

b.  Team  size 

Number  of  people  to  perform  the  tasks 


Level  or  Degree 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  Low 

o 

Very  Negative 

o 

Low 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

High 

o 

Positive 

o 

Very  High 

o 

Very  Positive 

o 
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c.  Team  structure 

Distribution  of  tasks  and  roles  between  the 
team  members 


Level  or  Degree 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  Low 

o 

Very  Negative 

o 

Low 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

High 

o 

Positive 

o 

Very  High 

o 

Very  Positive 

o 

28.  TEAM  CHARACTERISTICS 

a.  Team  maturity 

Factors  related  to  experience  in  working 
together  as  a  team 


Level  or  Degree 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  Low 

o 

Very  Negative 

o 

Low 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

High 

o 

Positive 

o 

Very  High 

o 

Very  Positive 

o 

b.  Face-to-face  interaction 

Time  spent  working  face-to-face  as  a  team 


Level  or  Degree 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  Low 

o 

Very  Negative 

o 

Low 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

High 

o 

Positive 

o 

Very  High 

o 

Very  Positive 

o 

c.  Match  between  team  goals  and 
organisational  goals 

Team  goals  fit  within  the  goals  of  the 
organisation 


Level  or  Degree 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  Low 

o 

Very  Negative 

o 

Low 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

High 

o 

Positive 

o 

Very  High 

o 

Very  Positive 

o 

29.  MANAGING  INFORMATION 

Obtaining,  processing,  and  exchanging 
information 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 
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ORGANIZATION 


a.  Obtaining  information 

Actively  searching  to  compensate  for  lack 
of  information 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 

b.  Processing  information 

Identifying  relevant  information  and 
integrating  it  with  existing  knowledge 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 

c.  Exchanging  information 

Timely,  clear,  concise,  correct,  complete 
communication 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 

d.  Assessing  the  situation 

Perceiving  and  understanding  the  situation, 
and  anticipating  events 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 

30.  DECISION  MAKING 

Understanding  the  problem,  evaluation  options 
and  estimating  effects  in  a  timely  manner 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 
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a.  Determining  the  exact  problem 

Analysing  the  critical  information  with  the 
right  expertise 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 

b.  Making  decisions  in  the  time  available 

Staying  aware  of  time  limits  and  adapting 
appropriately 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 

c.  Evaluating  options  and  results 

Considering  different  options  and  results, 
and  changing  decisions  if  necessary 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 

31.  PLANNING 

Determining  actions  necessary  to  achieve  the 
goals 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 

a.  Identifying  future  tasks  and  events 

Anticipating  upcoming  tasks  and  possible 
events 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 
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ORGANIZATION 


b.  Scheduling 

Developing  a  logical  timeline  of  the 
operation 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 

c.  Determining  the  required  resources 

Identifying  the  required  personnel,  equipment, 
and  preparation  time  for  the  operation 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 

d.  Defining  the  approach  to  accomplish 
the  mission 

Developing  operational  policies  and 
procedures 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 

32.  CARRYING  OUT  THE  PLANNED  TASKS 


Organising,  managing,  and  monitoring  the 
execution  of  the  plans 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 

a.  Organising 

Structuring  and  assigning  tasks,  and 
coordinating  team  activities 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 
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b.  Managing 

Prioritising  and  adjusting  tasks  and  resources 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 

c.  Monitoring  progress 

Ensuring  the  team  is  on  the  right  path  to 
achieving  the  goals 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 

33.  INTERACTIONS  WITH  OTHER  COMMAND  TEAMS 


Building  relationships  and  interacting  with  other 
teams 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 

34.  PROVIDING  AND  MAINTAINING  VISION 


Communication  of  the  team ’s  direction  and 
purpose 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 

35.  MAINTAINING  COMMON  INTENT 


Preserving  a  shared  sense  of  what  should  happen 
and  how 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 
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ANNEX  B  -  THE  CTEF  INSTRUMENT 

BASED  ON  CTEF  2005  MODEL  AS  EMPLOYED 

IN  THE  NATO  HFM  TG-127  MULTI-NATIONAL  SURVEY  STUDY 


ORGANIZATION 


36.  COLLABORATION  AMONG  TEAM  MEMBERS 


Communicating,  coordinating,  and  giving 
feedback  to  each  other 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 

a.  Team  communication 

Open  communication  and  expression  of 
thoughts 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 

b.  Team  coordination 

Coordination  of  each  others  activities  and 
way  of  working 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 

c.  Providing  feedback 

Providing  advice  to  each  other  to  improve 
performance 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 

36.  MOTIVATING 


Being  inspired  to  put  forth  greater  effort  to  reach 
the  goals 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 
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ANNEX  B  -  THE  CTEF  INSTRUMENT 
BASED  ON  CTEF  2005  MODEL  AS  EMPLOYED 
IN  THE  NATO  HFM  TG-127  MULTI-NATIONAL  SURVEY  STUDY 


a.  Rewarding  and  recognizing  in  your 
team 

Praising  and  rewarding  each  others 
contributions 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 

b.  Developing  interest  and  pride  in  the 
work 

Caring  about  the  work  itself 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 

37.  ADAPTING  TO  CHANGES 

Circumstances  outside  and  inside  the  team 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 

a.  Monitoring  team  members’  performance 

Observing  and  assessing  your  own  and 
other ’s  actions 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 

b.  Correcting  each  other 

Informing  team  members  when  they  make  a 
mistake 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 
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BASED  ON  CTEF  2005  MODEL  AS  EMPLOYED 

IN  THE  NATO  HFM  TG-127  MULTI-NATIONAL  SURVEY  STUDY 


ORGANIZATION 


c.  Providing  back-up 

Supporting  team  members  when  they  are 
busy 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 

38.  MAINTAINING  TEAM  SYNERGY 

Improving  or  maintaining  team  coherence 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 

a.  Providing  social  support  and  respect 

Providing  encouragement,  compliments, 
and  companionship 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 

b.  Regulating  emotions 

Preserving  emotional  balance  and  composure 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 

c.  Maintaining  cohesion 

Maintaining  unity,  solidarity,  and  team  spirit 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 
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d.  Handling  conflicts 

Solving  and  decreasing  arguments  and 
friction 


How  did  the  team  perform? 

Impact  on 

Overall  Team  Effectiveness 

Not  Applicable 

o 

Not  Applicable 

o 

Very  poorly 

o 

Very  Negative 

o 

Poorly 

o 

Negative 

o 

Moderate 

o 

No  Impact 

o 

Well 

o 

Positive 

o 

Very  well 

o 

Very  Positive 

o 

39.  TASK  OUTCOMES 

a.  Meeting  the  goals  of  the  commander  or 
of  higher  echelons 

Timeliness  and  completeness  of  objectives 
and  goals 


How  did  the  team  perform? 

Not  Applicable 
Very  poorly 
Poorly 

Moderate 

Well 

Very  well 

o 

o 

o 

o 

o 

o 

b.  Meeting  the  expectations  of  other  partners 

Expectations  of  local  authorities,  local 
population,  coalition  partners,  the  home 
front 


How  did  the  team  perform? 

Not  Applicable 
Very  poorly 
Poorly 

Moderate 

Well 

Very  well 

o 

o 

o 

o 

o 

o 

c.  Staying  within  limits  and  intentions  of 
the  mission 

Remaining  in  line  with  intentions,  orders, 
ndes-of- engagement,  ethical  standards 


How  did  the  team  perform? 

Not  Applicable 
Very  poorly 
Poorly 

Moderate 

Well 

Very  well 

o 

o 

o 

o 

o 

o 

40.  TEAM  OUTCOMES 

a.  Trust  among  team  members 

Belief  in  each  others  capacities,  loyalty  and 
dedication 


How  did  the  team  perform? 

Not  Applicable 
Very  poorly 
Poorly 

Moderate 

Well 

Very  well 

o 

o 

o 

o 

o 

o 
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b.  Morale 

Positive  attitude  towards  team  and  assignment, 
and  willing  to  put  effort  in  achieving  goals 


How  did  the  team  perform? 

Not  Applicable 
Very  poorly 
Poorly 

Moderate 

Well 

Very  well 

o 

o 

o 

o 

o 

o 

c.  Cohesion 

Feeling  attracted  towards  the  team  and  acting 
as  a  team  as  opposed  to  a  group  of  individuals 


How  did  the  team  perform? 

Not  Applicable 
Very  poorly 
Poorly 

Moderate 

Well 

Very  well 

o 

o 

o 

o 

o 

o 

d.  Confidence  that  the  team  will  successfully 
achieve  its  goals 

Team  members  believe  that  the  team  is  effective 


How  did  the  team  perform? 

Not  Applicable 
Very  poorly 
Poorly 

Moderate 

Well 

Very  well 

o 

o 

o 

o 

o 

o 

e.  Shared  vision  on  the  goals 

Team  members  have  a  similar  vision  on  the 
goals  the  team  has  to  achieve 


How  did  the  team  perform? 

Not  Applicable 
Very  poorly 
Poorly 

Moderate 

Well 

Very  well 

o 

o 

o 

o 

o 

o 

f.  Respect  among  team  members 

Team  members  try  to  understand  each  other, 
despite  their  differences 


How  did  the  team  perform? 

Not  Applicable 
Very  poorly 
Poorly 

Moderate 

Well 

Very  well 

o 

o 

o 

o 

o 

o 

41.  TASK  OUTCOMES:  REVIEWS  AND  ADJUSTMENTS 


a.  Reviewing  progress  and  achievement  of 
task  milestones  and  goals 

(In)formal  assessment  with  the  team  to  assure 
task  goals  are  achieved 


How  did  the  team  perform? 

Not  Applicable 
Very  poorly 
Poorly 

Moderate 

Well 

Very  well 

o 

o 

o 

o 

o 

o 
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b.  Taking  measures  to  improve  task 
processes  when  needed 

Improving  managing  information,  decision 
making,  planning  and  execution  of  tasks 


How  did  the  team  perform? 

Not  Applicable 
Very  poorly 
Poorly 

Moderate 

Well 

Very  well 

o 

o 

o 

o 

o 

o 

c.  Taking  measures  to  improve  task 
conditions  when  needed 

Improving  workload,  goal  clarity,  command 
structure,  organisational  support 


How  did  the  team  perform? 

Not  Applicable 
Very  poorly 
Poorly 

Moderate 

Well 

Very  well 

o 

o 

o 

o 

o 

o 

42.  TEAM  OUTCOMES:  REVIEWS  AND  ADJUSTMENTS 


a.  Reviewing  progress  and  achievement  of 
team  milestones  and  goals 

(In)formal  assessment  with  the  team  to  assure 
task  goals  are  achieved 


How  did  the  team  perform? 

Not  Applicable 
Very  poorly 
Poorly 

Moderate 

Well 

Very  well 

o 

o 

o 

o 

o 

o 

b.  Taking  measures  to  improve  team 
processes  when  needed 

Improving  maintaining  vision  and  intent, 
motivation,  working  together,  team  synergy’ 


How  did  the  team  perform? 

Not  Applicable 
Very  poorly 
Poorly 

Moderate 

Well 

Very  well 

o 

o 

o 

o 

o 

o 

c.  Taking  measures  to  improve  team 
conditions  when  needed 

Improving  personnel  and  leader  skills, 
expertise,  team  characteristics 


How  did  the  team  perform? 

Not  Applicable 
Very  poorly 
Poorly 

Moderate 

Well 

Very  well 

o 

o 

o 

o 

o 

o 

You  have  now  completed  the  CTEF  questionnaire.  The  following  questions  ask 
your  opinion  about  the  questionnaire  itself.  Please  indicate  to 
what  extent  you  agree  with  the  following  statements. 
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43. 1  think  that  the  right  aspects  of  team  effectiveness  are  being  addressed: 


Completely  Disagree 

Disagree 

Neutral 

Agree 

Completely  Agree 

0 

0 

0 

0 

0 

44. 1  did  not  notice  an  important  aspect,  namely... 


45. 1  think  that  the  questions  are  clear: 


Completely  Disagree 

Disagree 

Neutral 

Agree 

Completely  Agree 

0 

0 

0 

0 

0 

46. 1  think  that  the  distinction  between  level  and  impact  is  clear: 


Completely  Disagree 

Disagree 

Neutral 

Agree 

Completely  Agree 

0 

0 

0 

0 

0 

47. 1  think  that  the  distinction  between  level  and  impact  is  useful: 


Completely  Disagree 

Disagree 

Neutral 

Agree 

Completely  Agree 

0 

0 

o 

o 

0 

48. 1  think  that  the  length  of  the  questionnaire  is: 


Completely  Disagree 

Disagree 

Neutral 

Agree 

Completely  Agree 

0 

0 

0 

0 

0 

49. 1  learned  something  about  improving  team  effectiveness: 


Completely  Disagree 

Disagree 

Neutral 

Agree 

Completely  Agree 

0 

0 

o 

O 

0 

50. 1  would  be  willing  to  use  the  instrument  in  practice: 


Completely  Disagree 

Disagree 

Neutral 

Agree 

Completely  Agree 

0 

0 

o 

o 

0 

51.  Do  you  have  any  additional  comments  or  questions? 


Thanks  for  filling  in  this  questionnaire! 
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Annex  C  -  DEMOGRAPHIC  FREQUENCIES  FOR 
PARTICIPANTS  IN  THE  CTEF  SURVEY  STUDY 


Table  C-1:  Age  Distribution  of  Participants  in  the  CTEF  Survey  Study 


Age 

Frequency 

Percent 

Age 

Frequency 

Percent 

26 

5 

0.70 

41 

31 

4.32 

27 

4 

0.56 

42 

41 

5.71 

28 

7 

0.97 

46 

31 

4.32 

29 

12 

1.67 

47 

32 

4.46 

30 

5 

0.70 

48 

34 

4.74 

31 

13 

1.81 

49 

35 

4.87 

32 

6 

0.84 

50 

34 

4.74 

33 

17 

2.37 

51 

13 

1.81 

34 

19 

2.65 

52 

13 

1.81 

35 

25 

3.48 

53 

13 

1.81 

36 

19 

2.65 

54 

12 

1.67 

37 

40 

5.57 

55 

9 

1.25 

38 

30 

4.18 

56 

4 

0.56 

39 

36 

5.01 

57 

3 

0.42 

40 

31 

4.32 

58 

3 

0.42 

43 

40 

5.57 

59 

1 

0.14 

44 

58 

8.08 

60 

2 

0.28 

45 

40 

5.57 

Total 

718 

100.00 
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Table  C-2:  Nationality  Distribution  of  Participants  in  the  CTEF  Survey  Study 


Country 

Frequency 

Percent 

Belgium 

113 

15.74 

Bulgaria 

21 

2.92 

Canada 

338 

47.08 

Czech  Republic 

1 

0.14 

Denmark 

1 

0.14 

France 

2 

0.28 

Germany 

1 

0.14 

Netherlands 

30 

4.18 

Poland 

1 

0.14 

Russia 

1 

0.14 

Spain 

1 

0.14 

Switzerland 

1 

0.14 

United  Kingdom 

1 

0.14 

United  States 

198 

27.58 

Total 

710 

98.89 

System 

8 

1.11 

Table  C-3:  NATO  Equivalent  Rank  Distribution  of  Participants  in  the  CTEF  Survey  Study 


Rank 

Frequency 

Percent 

OF-8  (LGen) 

1 

0.14 

OF-6  (BGen) 

1 

.014 

OF-5  (Col) 

64 

8.91 

OF-4  (LCol) 

141 

19.64 

OF-3  (Maj) 

297 

41.36 

OF-2  (Capt) 

97 

13.51 

OF-1  (Lt) 

5 

.70 

OR-9  (MWO) 

37 

5.15 

OR- 8  (CWO) 

73 

10.17 

OR-7  (WO) 

1 

.014 

Total 

717 

99.99 

System 

1 

.01 
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Table  C-4:  Echelon  Distribution  of  Participants  in  the  CTEF  Survey  Study 


Echelon 

Frequency 

Percent 

CJTF  -  JFC  -  DJTF 

157 

21.87 

Corps  -  Fleet  -  Numbered  Air  Force 

36 

5.01 

Army  Division  -  Battle  Group 

60 

8.36 

Brigade  -  Ship  CO  -  Wing 

128 

17.83 

Battalion  -  Department  -  Squadron 

204 

28.41 

Company  -  Navy  Division  -  Flight 

130 

18.11 

Total 

715 

99.58 

System 

3 

0.42 

Table  C-5:  Services  of  Other  Command  Teams  Members 


Frequency 

Percent  of 
Responses 

Percent 
of  Cases 

Army 

606 

42.20 

84.52 

Air  Force 

365 

25.42 

50.91 

Navy 

306 

21.31 

42.68 

Marine  Corps 

89 

6.20 

12.41 

Coast  Guard 

2 

0.14 

0.28 

Federal  Civilian 

44 

3.06 

6.14 

Medical  Service 

11 

0.77 

1.53 

Special  Operations  Force 

5 

0.35 

0.70 

State  /  Local  /  Other  Government 

8 

0.56 

1.12 

Total 

1436 

100.00 

200.28 
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Table  C-6:  Nationalities  of  Other  Command  Teams  Members 


All  Nations 

N 

Percent 

%  of  Cases 

Albania 

11 

0.43 

1.54 

Armenia 

7 

0.27 

0.98 

Austria 

21 

0.81 

2.93 

Azerbaijan 

3 

0.12 

0.42 

Belgium 

162 

6.27 

22.63 

Bosnia  and  Herzegovina 

21 

0.81 

2.93 

Bulgaria 

35 

1.35 

4.89 

Croatia 

48 

1.86 

6.70 

Czech  Republic 

44 

1.70 

6.15 

Denmark 

71 

2.75 

9.92 

Estonia 

23 

0.89 

3.21 

Finland 

23 

0.89 

3.21 

France 

98 

3.79 

13.69 

Georgia 

8 

0.31 

1.12 

Germany 

114 

4.41 

15.92 

Greece 

49 

1.90 

6.84 

Hungary 

34 

1.32 

4.75 

Iceland 

7 

0.27 

0.98 

Ireland 

15 

0.58 

2.09 

Italy 

85 

3.29 

11.87 

Kazakhstan 

7 

0.27 

0.98 

Fatvia 

14 

0.54 

1.96 

Fithuania 

18 

0.70 

2.51 

Fuxembourg 

17 

0.66 

2.37 

Moldova 

1 

0.04 

0.14 

Montenegro 

2 

0.08 

0.28 

Netherlands 

141 

5.46 

19.69 

Poland 

66 

2.55 

9.22 

Norway 

69 

2.67 

9.64 

Portugal 

48 

1.86 

6.70 

Republic  of  Macedonia 

10 

0.39 

1.40 

Romania 

44 

1.70 

6.15 
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ANNEX  C  -  DEMOGRAPHIC  FREQUENCIES 
FOR  PARTICIPANTS  IN  THE  CTEF  SURVEY  STUDY 


All  Nations 

N 

Percent 

%  of  Cases 

Russia 

10 

0.39 

1.40 

Serbia 

4 

0.15 

0.56 

Slovakia 

26 

1.01 

3.63 

Slovenia 

20 

0.77 

2.79 

Spain 

54 

2.09 

7.54 

Sweden 

26 

1.01 

3.63 

Switzerland 

11 

0.43 

1.54 

Tajikistan 

2 

0.08 

0.28 

Turkey 

53 

2.05 

7.40 

Turkmenistan 

3 

0.12 

0.42 

Ukraine 

14 

0.54 

1.96 

United  Kingdom 

181 

7.00 

25.28 

Uzbekistan 

2 

0.08 

0.28 

Canada 

393 

15.21 

54.89 

United  States 

327 

12.65 

45.67 

Afghanistan 

7 

0.27 

0.98 

Argentina 

2 

0.08 

0.28 

Australia 

28 

1.08 

3.91 

Bahrain 

1 

0.04 

0.14 

Bangladesh 

3 

0.12 

0.42 

Bolivia 

3 

0.12 

0.42 

Brazil 

3 

0.12 

0.42 

Burundi 

2 

0.08 

0.28 

Cameroon 

1 

0.04 

0.14 

Chile 

4 

0.15 

0.56 

China 

1 

0.04 

0.14 

Colombia 

1 

0.04 

0.14 

Ecuador 

1 

0.04 

0.14 

Egypt 

4 

0.15 

0.56 

El  Salvador 

1 

0.04 

0.14 

Fiji 

1 

0.04 

0.14 

Ghana 

3 

0.12 

0.42 

India 

6 

0.23 

0.84 
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ANNEX  C  -  DEMOGRAPHIC  FREQUENCIES 
FOR  PARTICIPANTS  IN  THE  CTEF  SURVEY  STUDY 


ORGANIZATION 


All  Nations 

N 

Percent 

%  of  Cases 

Iraq 

9 

0.35 

1.26 

Jamaica 

1 

0.04 

0.14 

Japan 

6 

0.23 

0.84 

Jordan 

3 

0.12 

0.42 

Kenya 

3 

0.12 

0.42 

Kuwait 

1 

0.04 

0.14 

Nepal 

2 

0.08 

0.28 

New  Zealand 

8 

0.31 

1.12 

Oman 

1 

0.04 

0.14 

Pakistan 

9 

0.35 

1.26 

Paraguay 

2 

0.08 

0.28 

Peru 

2 

0.08 

0.28 

Philippines 

2 

0.08 

0.28 

Qatar 

1 

0.04 

0.14 

Republic  of  Korea 

7 

0.27 

0.98 

Rwanda 

3 

0.12 

0.42 

Saudi  Arabia 

1 

0.04 

0.14 

Sri  Lanka 

1 

0.04 

0.14 

United  Arab  Emirates 

1 

0.04 

0.14 

Uruguay 

3 

0.12 

0.42 

Zambia 

2 

0.08 

0.28 

Unknown  Country 

2 

0.08 

0.28 

Totals 

2584 

100.00 

360.89 
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Annex  D  -  CTEF  2.0  CONCEPTS 


ORGANIZATION 


CONDITIONS 

Mission  Context 

Situational  uncertainty 

Lack  of  goal  stability 

Task  Characteristics 

Task  complexity 

Workload 

Organisation 

Clarity  of  command  structure 
Organisational  support 

Match  of  team’s  mission  and 
organisational  goal 

Team  Leader 

Team  leader  competencies 

Match  of  personal  goals  to 
organisational  goals 

Team  Member 

Team  member  competencies 

Match  of  personal  goals  to 
organisational  goals 

Team 

Mix  of  people 

Team  structure 

Team  maturity 

Match  of  team  goals  to  organisational 
goals 

PROCESSES 

Task-Focused  Behaviours 

Managing  information 

Decision  making 

Planning 

Executing  plans 

Interacting  with  other  command  teams 

Team-Focused  Behaviours 

Providing  vision  and  intent 

Collaborating  between  team  members 
Motivating 

Monitoring  team  member’s  behaviours 
Providing  back-up 

Maintaining  team  cohesion 

OUTCOMES 

Task  Outcomes 

Meeting  the  goals  of  the  commander 
and  higher  echelons 

Staying  within  the  limits  and  intentions 
of  the  mission 

Team  Outcomes 

Trust  between  team  members 

Collective  confidence  in  achieving 
goals 

FEEDBACK 

Did  your  team  take  measures  to  improve  task  processes  when  needed? 

Did  your  team  take  measures  to  improve  team  processes  when  needed? 

RTO-TR-HFM-127 


D  -1 


ANNEX  D  -  CTEF  2.0  CONCEPTS 


D  -2 


RTO-TR-HFM-127 


Annex  E  -  CTEF  2.0  INSTRUMENT 


ORGANIZATION 


E.l  SECTION  1  -  THE  CONDITIONS 


MISSION  CONTEXT 

Characteristics  of  the  operational  environment 

Situation  uncertainty  is  ... 

Very  Low 

Low 

Moderate 

High 

Very  High 

Goal  instability  is  ... 

Very  Low 

Low 

Moderate 

High 

Very  High 

TASK  CHARACTERISTICS 

Characteristics  of  the  team’s  tasks 

Task  complexity  is  ... 

Very  Low 

Low 

Moderate 

High 

Very  High 

Workload  is  ... 

Very  Low 

Low 

Moderate 

High 

Very  High 

ORGANISATION 

Characteristics  of  the  organisation 

Match  of  the  organizational  goal 
and  the  team’s  mission  is  ... 

Very  low 

Low 

Moderate 

High 

Very  High 

Clarity  of  command  structure  is  ... 

Very  low 

Low 

Moderate 

High 

Very  High 

The  support  by  the  organisation  is  ... 

Very  low 

Low 

Moderate 

High 

Very  High 

TEAM 

Characteristics  of  the  group  of  people  you  work  with  directly 

Mix  of  people  is  ... 

Very 

Inappropriate 

Inappropriate 

Moderate 

Appropriate 

Very 

Appropriate 

Distribution  of  tasks  and  roles  is  ... 

Very 

Inappropriate 

Inappropriate 

Moderate 

Appropriate 

Very 

Appropriate 

Team  maturity  is  ... 

Very  Low 

Low 

Moderate 

High 

Very  High 

Match  of  the  team’s  own  goals  to 
the  organizational  goals  is  ... 

Very  Low 

Low 

Moderate 

High 

Very  High 

TEAM  LEADER 

Characteristics  of  the  leader  you  work  with  directly 

Team  leader  competencies  are  ... 

Very  Low 

Low 

Moderate 

High 

Very  High 

Match  of  the  team  leader  personal  goals 
to  the  organizational  goals  is  ... 

Very  Low 

Low 

Moderate 

High 

Very  High 
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ORGANIZATION 


TEAM  MEMBER 

Characteristics  of  the  team  members  you  work  with  directly 

Team  member  competencies  are  ... 

Very  Low 

Low 

Moderate 

High 

Very  High 

Match  of  team  member’s  personal  goals 
to  the  organizational  goals  is  ... 

Very  Low 

Low 

Moderate 

High 

Very  High 

E.2  SECTION  2  -  THE  PROCESSES 


TASK-FOCUSED  BEHAVIOURS 

Characteristics  of  the  task  behaviours 

Managing  information  is  done  ... 

Very 

Poorly 

Poorly 

Moderate 

Well 

Very 

Well 

Decision  making  is  done  ... 

Very 

Poorly 

Poorly 

Moderate 

Well 

Very 

Well 

Planning  is  done  ... 

Very 

Poorly 

Poorly 

Moderate 

Well 

Very 

Well 

Executing  plans  is  done  ... 

Very 

Poorly 

Poorly 

Moderate 

Well 

Very 

Well 

Interacting  with  other  command  teams  is  done  ... 

Very 

Poorly 

Poorly 

Moderate 

Well 

Very 

Well 

TEAM-FOCUSED  BEHAVIOURS 

Characteristics  of  the  team  behaviours 

Providing  vision  and  intent  is  done  ... 

Very 

Poorly 

Poorly 

Moderate 

Well 

Very 

Well 

Collaborating  between  team  members  is  done  ... 

Very 

Poorly 

Poorly 

Moderate 

Well 

Very 

Well 

Motivating  is  done  ... 

Very 

Poorly 

Poorly 

Moderate 

Well 

Very 

Well 

Monitoring  team  member’s  behaviours  is  done  ... 

Very 

Poorly 

Poorly 

Moderate 

Well 

Very 

Well 

Providing  back-up  is  done  ... 

Very 

Poorly 

Poorly 

Moderate 

Well 

Very 

Well 

Maintaining  team  cohesion  is  done  ... 

Very 

Poorly 

Poorly 

Moderate 

Well 

Very 

Well 

E.3  SECTION  3  -  THE  OUTCOMES 


TASK  OUTCOMES 

Intermediate  or  final  task  achievements 

Meeting  the  goals  of  the  commander  and 
higher  echelons  is  achieved  ... 

Very 

poorly 

Poorly 

Moderate 

Well 

Very 

Well 

Staying  within  the  limits  and  intentions 
of  the  mission  is  achieved  ... 

Very 

poorly 

Poorly 

Moderate 

Well 

Very 

Well 
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TEAM  OUTCOMES 

Intermediate  or  final  team  achievements 

Trust  between  team  members  is  achieved  ... 

Very 

Poorly 

Poorly 

Moderate 

Well 

Very 

Well 

Collective  confidence  in  reaching 
goals  is  achieved  ... 

Very 

Poorly 

Poorly 

Moderate 

Well 

Very 

Well 

E.4  SECTION  4  -  FEEDBACK 


FEEDBACK 

Review  of  achievement  may  need  measures  to  improve  task  and  team  behaviours 

Taking  measures  to  improve  task 
processes  was  done  ... 

Very 

Poorly 

Poorly 

Moderate 

Well 

Very 

Well 

Taking  measures  to  improve  team 
processes  was  done  ... 

Very 

Poorly 

Poorly 

Moderate 

Well 

Very 

Well 
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Annex  F  -  PRACTICAL  GUIDELINES 


Practical  guidelines  for  successful  use  of  the  questionnaire  will  be  described  in  the  following  paragraphs. 
The  following  subjects  will  be  described: 

•  What  teams  are  suitable  for  assessment? 

•  What  preparations  are  needed  before  using  the  questionnaire? 

•  How  results  can  be  analyzed  and  interpreted. 

Section  1.01  Teams  Suitable  for  Assessment 

The  instrument  is  developed  for  use  by  military  commanders  and  their  staff,  and  by  commanders  and  his/her 
sub-commanders  at  different  levels  (for  example  battalion  level  and  brigade  level,  ships).  With  some 
adjustments  the  instrument  can  also  be  used  for  civilian  teams,  like  emergency  teams.  Team  effectiveness 
can  be  measured  when  teams  perform  different  types  of  tasks,  for  example  planning,  command  and  control, 
and  staff  processes.  Small  and  large  teams  can  be  assessed  with  this  questionnaire.  It  is  possible  that  people 
can  be  a  member  of  more  than  one  team.  For  example,  the  head  of  a  section  is  a  member  of  the  section  team, 
but  is  also  a  member  of  the  staff  team.  The  team  that  will  be  assessed  has  to  be  defined  prior  to  assessment, 
so  that  it  is  clear  for  the  respondents  regarding  which  team  the  questions  have  to  be  answered. 

Section  1.02  Preparations  Prior  to  Assessment 

Before  distributing  the  questionnaire  some  preparations  have  to  be  made.  These  are  described  here: 

1)  Adjustments  of  the  Questionnaire: 

•  Replace  for  each  item  the  word  ‘team’  with  the  specific  name  of  the  team,  for  example  battle  group 
staff.  By  doing  so  it  is  clear  for  the  team  members  regarding  which  team  the  questions  have  to  be 
answered. 

•  Check  if  all  the  items  are  appropriate  for  the  (educational/training/mission)  situation  and  the  team 
that  is  assessed.  What  are  the  goals  of  the  training,  the  tasks  of  the  team  and  what  is  the  scenario? 
Items  can  be  adjusted  for  the  specific  operational  context  and  team. 

•  Terms  that  are  used  in  the  questionnaire  have  to  be  univocal  and  clearly  understood.  Make  sure 
that  all  terms  are  understood  in  the  same  way  by  all  team  members. 

2)  Computer-Based  Questionnaire: 

Assess  the  feasibility  of  computer-based  administration  of  the  questionnaire.  The  advantage  is  that  data 
can  be  collected  easily  and  that  results  can  be  analyzed  more  quickly.  Pre-test  the  questionnaire  to  confirm 
that  the  automated  data  collection  works  properly,  especially  if  changes  are  made  to  the  questionnaire. 
If  the  computer-based  questionnaire  does  not  work  properly  it  will  frustrate  respondents  and  reduce  the 
number  of  valid  cases. 

3)  Moment  of  Assessment: 

Select  carefully  the  moment  of  assessment.  The  selected  time  influences  not  only  the  number  of  responses 
but  also  the  quality.  Use  the  training  schedule  to  select  an  appropriate  moment. 

4)  Guarantee  Anonymity: 

Make  sure  that  anonymity  of  the  team  members  who  fill  in  the  questionnaire  is  guaranteed.  Think  about 
how  this  can  be  achieved,  because  it  influences  both  the  response  rate  and  the  quality  of  the  answers  that 
people  give. 
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ORGANIZATION 


5)  Analysis  of  Results: 

Determine  in  advance  what  criteria  will  be  used  to  interpret  the  results  of  the  questionnaires  (see  Section 
1.04  below). 

6)  Timing  of  Feedback/Discussion: 

Plan  in  advance  a  suitable  time  to  present  the  results  of  the  questionnaire  and  who  has  to  be  present. 
Discussion  of  the  results  and  feedback  can  be  a  part  of  the  after  action  review.  But  also  a  different  moment 
can  be  selected. 


Section  1.03  Assessment 

As  example  see  the  questionnaire  used  for  the  survey  in  Annex  B.  Stress  the  following  points  to  the  team 
that  has  to  fill  in  the  questionnaire: 

•  Purpose  of  the  questionnaire; 

•  The  assessment  is  not  an  individual  assessment,  but  measures  effectiveness  of  the  team  as  a  whole; 

•  The  data  will  be  analyzed  anonymously; 

•  No  individual  results  will  be  presented; 

•  Instruction  how  to  fill  out  the  questionnaire;  and 

•  Filling  out  the  questionnaire  takes  approximately  10-15  minutes. 

Section  1.04  Analysis  of  Results 

It  is  important  to  decide  in  advance  what  the  level  of  performance  of  the  team  must  be.  Depending  on  the 
ambition  level  of  the  commander  a  certain  percentage  can  be  chosen  as  a  criterion,  see  Table  F-l.  For  a  more 
experienced  team,  for  example,  a  high  criterion  might  be  chosen  (i.e.,  lower  percentages  in  the  low 
categories).  When  the  questionnaire  is  filled  out,  for  each  item  the  percentages  of  the  respondents  that 
chose  a  certain  category  can  be  calculated.  If  a  category  exceeds  the  chosen  level,  the  content  of  this  item 
needs  action  for  improvement.  Positive  results  can  also  be  used  to  stimulate  good  performance  of  the 
team. 
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Table  F-1:  Setting  a  Criterion  Level  for  the  Team. 


Level  of  Ambition 

Total  %  of  Respondents 
of  Two  Lowest  Categories 

Action 

High:  20%  level 

Higher  than  20% 

Improve  performance 

Lower  than  20% 

Maintain  performance 

Moderate:  30%  level 

Higher  than  30% 

Improve  performance 

Lower  than  30% 

Maintain  performance 

Low:  40%  level 

Higher  than  40% 

Improve  performance 

Lower  than  40% 

Maintain  performance 

As  can  be  seen  in  Table  F-1,  depending  on  the  criterion  chosen,  different  items  will  be  marked  as 
insufficient.  In  this  example  two  different  criteria  are  used  for  the  same  data.  In  general,  a  higher  criterion 
will  not  necessarily  lead  to  more  items  that  are  unsatisfactory,  because  a  higher  criterion  will  be  chosen  for 
a  team  that  is  expected  to  perform  well.  Most  of  the  conditions  and  processes  will  be  judged  satisfactory, 
and  therefore  a  higher  criterion  is  needed  to  identify  what  aspects  still  can  be  improved. 


Table  F-2:  High  Level  of  Ambition:  20%  Level. 


Very  Low 

High 

Very  High 

Item  A 

60% 

20% 

Item  B 

20% 

72% 

Item  C 

50% 

18% 

Item  D 

3% 

92% 

Item  E 

35% 

5% 

Table  F-3:  Low  Level  of  Ambition:  40%  Level. 

Very  Low 

Low 

High 

Very  High 

Item  A 

5% 

15% 

60% 

20% 

Item  B 

3% 

5% 

20% 

72% 

Item  C 

12% 

20% 

50% 

18% 

Item  D 

2% 

3% 

3% 

92% 

Item  E 

35% 

5% 

The  criterion  helps  to  determine  which  factors  are  satisfactory  and  which  ones  need  improvement. 
However,  to  be  able  to  interpret  the  results  it  is  important  to  discuss  the  results  with  the  team  members. 
This  will  give  more  insight  in  why  certain  processes,  conditions  or  outcomes  were  not  satisfactory.  If  there 
is  disagreement  on  certain  factors,  than  a  discussion  helps  also  to  gain  more  understanding  of  each  others’ 
point  of  view.  More  understanding  will  be  gained  about  what  is  used  as  a  frame  of  reference  to  judge  the 
conditions,  processes  and  outcomes  and  how  the  questions  are  interpreted  by  the  different  team  members. 
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